You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org> on 2009/10/06 07:59:31 UTC
[jira] Created: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
MRReliability test does not work with retired jobs
--------------------------------------------------
Key: MAPREDUCE-1062
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.21.0
Reporter: Sreekanth Ramakrishnan
Assignee: Sreekanth Ramakrishnan
Currently the MRReliability uses job clients get all job api which also includes retired jobs.
In case we have retired jobs in cluster,
The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Status: Open (was: Patch Available)
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1062:
-------------------------------------
Release Note: Ensure that MRReliability works with retired-jobs feature turned on.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Status: Patch Available (was: Open)
Re running thro' Hudson
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Status: Open (was: Patch Available)
Cancelling patch and running thro' hudson with latest patch.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765572#action_12765572 ]
Hemanth Yamijala commented on MAPREDUCE-1062:
---------------------------------------------
Looks OK to me. Sreekanth, can you confirm running reliability test with the retired jobs feature on is working now ?
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765520#action_12765520 ]
Hadoop QA commented on MAPREDUCE-1062:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421948/mapreduce-1062-2.patch
against trunk revision 825055.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/72/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/72/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/72/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/72/console
This message is automatically generated.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-1062-4.patch
Attaching mapred-1062-3.patch as mapreduce-1062-4.patch for running thro' hudson.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Ramya R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762666#action_12762666 ]
Ramya R commented on MAPREDUCE-1062:
------------------------------------
This is a duplicate of MAPREDUCE-1053
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787421#action_12787421 ]
Hadoop QA commented on MAPREDUCE-1062:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427293/mapreduce-1062-4.patch
against trunk revision 888269.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/301/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/301/console
This message is automatically generated.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-1062-2.patch
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Status: Patch Available (was: Open)
Rerunning the patch thro' hudson.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-1062-1.patch
Attaching patch for 21 and trunk.
Using schedulers job list which always has only waiting/running jobs in it. in order which scheduler looks at it. Which makes the lastest job in last position.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764940#action_12764940 ]
Sreekanth Ramakrishnan commented on MAPREDUCE-1062:
---------------------------------------------------
Attached patch fixes to use {{jobsToComplete()}} instead of {{getJobsFromQueue()}}.
Modified the javadoc and usage to mention that tests should run on a free cluster.
Also failing nosily if the job returned from the {{jobsToComplete()}} is complete.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Status: Patch Available (was: Open)
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1062:
-------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
Resolution: Fixed
I committed this. Thanks, Sreekanth!
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.22.0
>
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-ydist-20-1.patch
Attaching Yahoo! distribution patch, tested it running test on a cluster with retired jobs.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-1062-3-ydist.patch
Attaching Yahoo! distribution patch. The trunk patch is blocked by HADOOP-6269 as the {{JobClient.jobsToComplete()}} on trunk creates new JobConf and job submission is happening at same time causing a {{ConcurrentModificationException}} due to {{defaultResources}} being modified in the {{Configuration}} object.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-1062:
----------------------------------------------
Attachment: mapreduce-1062-3.patch
Missed out Hemanth's comment about removing the {{prevJobNum}}. Also corrected typo in documentation based on Hemanth's comment.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764701#action_12764701 ]
Hemanth Yamijala commented on MAPREDUCE-1062:
---------------------------------------------
I started looking at the patch. Unfortunately, I think the current algorithm makes assumptions about how the scheduler works. So, while it works perfectly well for the CapacityTaskScheduler, it may not work correctly with the FairshareScheduler, because the latter removes jobs it maintains per pool lazily. Hence, there may be a case where the number of jobs returned by getJobsFromQueue is non-zero, but it doesn't mean the current job is submitted.
I think there is already an assumption that this test is run independently on a cluster, because it kills tasktrackers etc and could affect other jobs if they are run in parallel. For the same reason, jobs within the reliability test are run one after the other. So, wouldn't it be right to use jobsToComplete instead of getJobsFromQueue and as long as this is non-zero, we can assume it is the job most recently submitted ?
Some other minor points:
- Can we update the documentation to say how the reliability test should be run ? For instance, we have to run it on a cluster that is not running other jobs, as stated above.
- Also, I would suggest we fail noisily if the last job we get is not in the PREP or RUNNING state, so that we wouldn't have false positive runs of the MRReliabiliy test.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783091#action_12783091 ]
Hadoop QA commented on MAPREDUCE-1062:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422201/mapreduce-1062-3-ydist.patch
against trunk revision 884628.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/274/console
This message is automatically generated.
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1062) MRReliability test does not work
with retired jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858565#action_12858565 ]
Hudson commented on MAPREDUCE-1062:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #291 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/291/])
MAPREDUCE-1062. Fix ReliabilityTest to work with retired jobs. Contributed by Sreekanth Ramakrishnan
> MRReliability test does not work with retired jobs
> --------------------------------------------------
>
> Key: MAPREDUCE-1062
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Sreekanth Ramakrishnan
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.22.0
>
> Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-1062-4.patch, mapreduce-ydist-20-1.patch
>
>
> Currently the MRReliability uses job clients get all job api which also includes retired jobs.
> In case we have retired jobs in cluster,
> The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.