You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Matei Zaharia (JIRA)" <ji...@apache.org> on 2009/12/02 07:02:20 UTC
[jira] Created: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Fair scheduler event log not logging job info
---------------------------------------------
Key: MAPREDUCE-1258
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/fair-share
Affects Versions: 0.21.0
Reporter: Matei Zaharia
Assignee: Matei Zaharia
Priority: Minor
The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-1258:
-------------------------------------
Fix Version/s: 0.21.0
I've committed this to 0.21 as well, to let this logging feature continue to work as in 0.20.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792595#action_12792595 ]
Scott Chen commented on MAPREDUCE-1258:
---------------------------------------
+1, this patch looks good to me.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792768#action_12792768 ]
Matei Zaharia commented on MAPREDUCE-1258:
------------------------------------------
Thanks for the review, Scott. I'll wait to see if there are any checkstyle warnings etc and commit it if there aren't.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-1258:
-------------------------------------
Status: Patch Available (was: Open)
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-1258:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Status: Resolved (was: Patch Available)
The test failure is unrelated, and I didn't included unit tests because this is a small logging fix, so I've committed this patch. I'm also going to check whether it's possible to commit it to 0.21.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793293#action_12793293 ]
Matei Zaharia commented on MAPREDUCE-1258:
------------------------------------------
Dhruba let me know that it's fine to commit this to 0.21 too, because it's a bug fix. I'll wait a day to see if there are any objections, and otherwise I'll commit it.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793288#action_12793288 ]
dhruba borthakur commented on MAPREDUCE-1258:
---------------------------------------------
I vote that we pull this into 0.21 as well.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-1258:
-------------------------------------
Attachment: mapreduce-1258-1.patch
Here's a patch for this issue. I didn't include a unit test because it's a very simple fix. I'd appreciate if someone could review it!
Note that the code in the patch does not print deficits, unlike the previous code, because deficits were removed as part of MAPREDUCE-706.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796736#action_12796736 ]
Hudson commented on MAPREDUCE-1258:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not
logging job info
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792853#action_12792853 ]
Hadoop QA commented on MAPREDUCE-1258:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428407/mapreduce-1258-1.patch
against trunk revision 892411.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/console
This message is automatically generated.
> Fair scheduler event log not logging job info
> ---------------------------------------------
>
> Key: MAPREDUCE-1258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Minor
> Attachments: mapreduce-1258-1.patch
>
>
> The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.
> It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.