You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/10/28 09:51:59 UTC
[jira] Created: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Two log statements at INFO level fill up jobtracker logs
--------------------------------------------------------
Key: MAPREDUCE-1160
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Hemanth Yamijala
Assignee: Ravi Gummadi
There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
The two statements identified are the following:
INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
and
INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1160:
----------------------------------------
Status: Patch Available (was: Open)
+1 for Hong's patch. Running through Hudson.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-1160:
----------------------------------------
Affects Version/s: 0.20.3
0.20.1
0.20.2
Let's get this fixed in the 0.20 branch too please. :)
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1, 0.20.2, 0.20.3
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1160-20.patch, MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770880#action_12770880 ]
Hemanth Yamijala commented on MAPREDUCE-1160:
---------------------------------------------
+1.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771498#action_12771498 ]
Hudson commented on MAPREDUCE-1160:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #127 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/])
. Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1160-20.patch, MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1160:
----------------------------------------
Status: Open (was: Patch Available)
Canceling patch to take care of Hong's new patch.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "gary murry (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773187#action_12773187 ]
gary murry commented on MAPREDUCE-1160:
---------------------------------------
I agree that this does not need unit testing, because we are just changing the log level of the items, but we do need to test have tests for log entries when we can. We rely on logging to help debug issues in the wild and need to have assurance that they are working correctly. Thanks.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1160-20.patch, MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1160:
---------------------------------
Attachment: MR-1160-2.patch
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1160:
------------------------------------
Attachment: MR-1160.patch
Attaching patch by changing the log statements to debug level.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770994#action_12770994 ]
Hong Tang commented on MAPREDUCE-1160:
--------------------------------------
When testing mumak, we also found that a few LOG.debug() statements include unnecessary string concatenation operations. This is probably a good place to fix those too. I will upload a updated patch shortly.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770999#action_12770999 ]
Hong Tang commented on MAPREDUCE-1160:
--------------------------------------
Revision to my previous statement:
When testing mumak, we found 24% of the cpu time is spent in string concatenation operations in the LOG.debug() statement in ResourceEstimator.getEstimatedTotalMapOutputSize()
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770883#action_12770883 ]
Amar Kamat commented on MAPREDUCE-1160:
---------------------------------------
Ravi,
Can you please check JobTracker, JobInProgress and TaskInProgress for similar cases?
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771048#action_12771048 ]
Hadoop QA commented on MAPREDUCE-1160:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423427/MR-1160.patch
against trunk revision 830531.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/console
This message is automatically generated.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1160:
------------------------------------
Release Note: Changed some log statements that were filling up jobtracker logs to debug level.
Status: Patch Available (was: Open)
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771147#action_12771147 ]
Hadoop QA commented on MAPREDUCE-1160:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423467/MR-1160-2.patch
against trunk revision 830531.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/console
This message is automatically generated.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771283#action_12771283 ]
Hudson commented on MAPREDUCE-1160:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #101 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/101/])
. Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1160-20.patch, MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770890#action_12770890 ]
Amar Kamat commented on MAPREDUCE-1160:
---------------------------------------
+1
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770835#action_12770835 ]
Hemanth Yamijala commented on MAPREDUCE-1160:
---------------------------------------------
We can move these to debug level under an if (LOG.isDebugEnabled()) check.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1160:
----------------------------------------
Attachment: MAPREDUCE-1160-20.patch
Attaching a patch for Hadoop 0.20. (Not for commit).
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1160-20.patch, MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771270#action_12771270 ]
Hemanth Yamijala commented on MAPREDUCE-1160:
---------------------------------------------
The -1 from tests is expected because this is only a change to log lines and hence no tests were included. I will commit this patch now.
It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1160:
----------------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this to trunk. Thanks, Ravi and Hong !
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770885#action_12770885 ]
Ravi Gummadi commented on MAPREDUCE-1160:
-----------------------------------------
Checked JobTracker, JobInProgress and TaskInProgress. I don't see any other similar cases there.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1160) Two log statements at INFO level
fill up jobtracker logs
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771271#action_12771271 ]
Hemanth Yamijala commented on MAPREDUCE-1160:
---------------------------------------------
bq. It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that.
HADOOP-6341.
> Two log statements at INFO level fill up jobtracker logs
> --------------------------------------------------------
>
> Key: MAPREDUCE-1160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Attachments: MR-1160-2.patch, MR-1160.patch
>
>
> There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.
> The two statements identified are the following:
> INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
> and
> INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.