You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2012/05/04 23:10:48 UTC
[jira] [Created] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Ravi Prakash created MAPREDUCE-4227:
---------------------------------------
Summary: TimeWindow statistics are not updated for TaskTrackers which have been restarted.
Key: MAPREDUCE-4227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270040#comment-13270040 ]
Ravi Prakash commented on MAPREDUCE-4227:
-----------------------------------------
I ran test-patch:
{noformat}
-1 overall.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
{noformat}
The findbugs warning is spurious. We've been through this before https://issues.apache.org/jira/browse/MAPREDUCE-4088?focusedCommentId=13263990&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13263990 .
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4227.branch-1.patch
>
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278860#comment-13278860 ]
Hadoop QA commented on MAPREDUCE-4227:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525873/MAPREDUCE-4227.branch-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2396//console
This message is automatically generated.
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4227.branch-1.patch
>
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-4227:
------------------------------------
Attachment: MAPREDUCE-4227.branch-1.patch
Hi Matt,
I'm attaching the patch. It makes the following changes
1. buckets are removed only if there are more in the current TaskTrackers than are required
2. Earlier the TimeWindow statistics were not being updated asap. They would lag behind by one "update cycle" I've fixed that as well.
3. I took the liberty to add some much needed javadocs in this JIRA.
Disclaimer: StatisticsCollector is being referenced in some AspectJ files. Unfortunately I don't know how I can check if my changes affected anything in there. Any clues?
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4227.branch-1.patch
>
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-4227:
----------------------------------
Target Version/s: 1.1.0 (was: 1.0.3)
Hi, Ravi and Koji, could you please convert this issue into a patch? Thanks.
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-4227:
------------------------------------
Status: Patch Available (was: Open)
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4227.branch-1.patch
>
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280200#comment-13280200 ]
Ravi Prakash commented on MAPREDUCE-4227:
-----------------------------------------
bq. -1 patch. The patch command could not apply the patch.
Yeah! This patch applies only to branch-1
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4227.branch-1.patch
>
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4227) TimeWindow statistics are not
updated for TaskTrackers which have been restarted.
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268715#comment-13268715 ]
Ravi Prakash commented on MAPREDUCE-4227:
-----------------------------------------
I'd been banging my head on this bug for weeks and then Koji swept in like an angel that he is, and rescued me.
Courtesy Koji!
{quote}
src/mapred/org/apache/hadoop/mapred/StatisticsCollector.java
{noformat}
276 synchronized void update() {
277 updates++;
278 if (updates == updatesPerBucket) {
279 for(TimeStat stat : statToCollect.values()) {
280 stat.addBucket();
281 }
282 updates = 0;
283 buckets++;
284 if (buckets > collectBuckets) {
285 for (TimeStat stat : statToCollect.values()) {
286 stat.removeBucket();
287 }
288 buckets--;
289 }
290 }
291 }
292 }
{noformat}
'bucket' keeps track of how many buckets ALL the TimeStat has for that Range (Hourly,Daily) so after 1 hour, buckets == collectBuckets and each time it adds, it takes out 1.
i believe the incorrect assumption they are making here is ALL the TimeStat would have same number of buckets
{quote}
One of these days I'll start a Koji fan club! Please consider joining.
> TimeWindow statistics are not updated for TaskTrackers which have been restarted.
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4227
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4227
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
>
> Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira