You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2009/01/06 01:37:44 UTC
[jira] Created: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Job counters sometimes go down as tasks run without task failures
-----------------------------------------------------------------
Key: HADOOP-4983
URL: https://issues.apache.org/jira/browse/HADOOP-4983
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Owen O'Malley
Priority: Critical
As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4983:
--------------------------------------------
Attachment: patch-4983.txt
Patch with the fix
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.19.1
>
> Attachments: patch-4983.txt
>
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4983:
----------------------------------
Affects Version/s: 0.19.0
Fix Version/s: 0.19.1
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.19.1
>
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661457#action_12661457 ]
Amareshwari Sriramadasu commented on HADOOP-4983:
-------------------------------------------------
This happens, because TaskInProgress replaces counters in every status update and sometimes, status update doesnot have counters.
The code doing the same is in the method, TaskInProgress.recomputeProgress() :
{code}
} else if (status.getRunState() == TaskStatus.State.RUNNING) {
if (status.getProgress() >= bestProgress) {
bestProgress = status.getProgress();
bestState = status.getStateString();
bestCounters = status.getCounters();
}
}
}
{code}
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Priority: Critical
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-4983:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
0.20.0
0.18.3
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.18.3, 0.19.1, 0.20.0, 0.21.0
>
> Attachments: patch-4983.txt
>
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4983:
--------------------------------------------
Status: Patch Available (was: Open)
I monitored 3 long running jobs with long running tasks, whose counters oscillate over the run of the job, without the patch.
And with patch applied the jobs had incrementing counters, they never went down.
test-patch result :
{noformat}
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec]
{noformat}
It is not easy to write a test-case for this.
All core and contrib unit tests passed on my machine.
The same patch applies to 0.19, 0.20 and trunk
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.19.1
>
> Attachments: patch-4983.txt
>
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4983) Job counters sometimes go down as
tasks run without task failures
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-4983:
-------------------------------------
Assignee: Amareshwari Sriramadasu
> Job counters sometimes go down as tasks run without task failures
> -----------------------------------------------------------------
>
> Key: HADOOP-4983
> URL: https://issues.apache.org/jira/browse/HADOOP-4983
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.19.1
>
>
> As tasks run, the counters seem to back up and move forward again. They always seem to be right when the task completes. I suspect this may have been introduced in HADOOP-2208.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.