You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2009/03/16 12:40:51 UTC
[jira] Created: (HADOOP-5499) JobTracker metrics do not match job
counters
JobTracker metrics do not match job counters
--------------------------------------------
Key: HADOOP-5499
URL: https://issues.apache.org/jira/browse/HADOOP-5499
Project: Hadoop Core
Issue Type: Bug
Components: mapred, metrics
Affects Versions: 0.20.0
Reporter: Chris Douglas
After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
{noformat}
mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
{noformat}
After the second (w/ one manually killed reduce)
{noformat}
mapred.tasktracker: hostName=pineapple-lm, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=pineapple-lm, sessionId=, jobs_completed=2, jobs_submitted=2, \
maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
{noformat}
The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.
# The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup tasks as well
# After being recorded among launched tasks, it looks like setup and cleanup are _not_ included among completed tasks
# {{tasks_completed}} on the TaskTracker should include only user tasks
# The {{reduces_launched}} count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5499) JobTracker metrics do not match job
counters
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas resolved HADOOP-5499.
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Fixed in HADOOP-5514
> JobTracker metrics do not match job counters
> --------------------------------------------
>
> Key: HADOOP-5499
> URL: https://issues.apache.org/jira/browse/HADOOP-5499
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred, metrics
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
> {noformat}
> mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
> reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
> mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
> maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
> {noformat}
> After the second (w/ one manually killed reduce)
> {noformat}
> mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
> reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
> mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=2, jobs_submitted=2, \
> maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
> {noformat}
> The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.
> # The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup tasks as well
> # After being recorded among launched tasks, it looks like setup and cleanup are _not_ included among completed tasks
> # {{tasks_completed}} on the TaskTracker should include only user tasks
> # The {{reduces_launched}} count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5499) JobTracker metrics do not match job
counters
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5499:
----------------------------------
Description:
After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
{noformat}
mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
{noformat}
After the second (w/ one manually killed reduce)
{noformat}
mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=2, jobs_submitted=2, \
maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
{noformat}
The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.
# The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup tasks as well
# After being recorded among launched tasks, it looks like setup and cleanup are _not_ included among completed tasks
# {{tasks_completed}} on the TaskTracker should include only user tasks
# The {{reduces_launched}} count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?
was:
After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
{noformat}
mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
{noformat}
After the second (w/ one manually killed reduce)
{noformat}
mapred.tasktracker: hostName=pineapple-lm, sessionId=, mapTaskSlots=3, maps_running=0, \
reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=pineapple-lm, sessionId=, jobs_completed=2, jobs_submitted=2, \
maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
{noformat}
The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.
# The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup tasks as well
# After being recorded among launched tasks, it looks like setup and cleanup are _not_ included among completed tasks
# {{tasks_completed}} on the TaskTracker should include only user tasks
# The {{reduces_launched}} count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?
> JobTracker metrics do not match job counters
> --------------------------------------------
>
> Key: HADOOP-5499
> URL: https://issues.apache.org/jira/browse/HADOOP-5499
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred, metrics
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
>
> After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
> {noformat}
> mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
> reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
> mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
> maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
> {noformat}
> After the second (w/ one manually killed reduce)
> {noformat}
> mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
> reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
> mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=2, jobs_submitted=2, \
> maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
> {noformat}
> The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.
> # The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup tasks as well
> # After being recorded among launched tasks, it looks like setup and cleanup are _not_ included among completed tasks
> # {{tasks_completed}} on the TaskTracker should include only user tasks
> # The {{reduces_launched}} count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.