You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Oleksandr Shevchenko (JIRA)" <ji...@apache.org> on 2018/09/05 10:59:00 UTC
[jira] [Created] (MAPREDUCE-7134) Setup and cleanup progress
information are not available after job completion
Oleksandr Shevchenko created MAPREDUCE-7134:
-----------------------------------------------
Summary: Setup and cleanup progress information are not available after job completion
Key: MAPREDUCE-7134
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7134
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Oleksandr Shevchenko
Information about "setupProgress" and "cleanupProgress" are not available after a job completion. These properties are ignored during creating JobReport since they were not added to jhist file.
We can get this information during a job is running:
{code}
import org.apache.hadoop.mapred.JobClient
import org.apache.hadoop.conf.Configuration
val client = new JobClient(new Configuration())
val job = client.getJob("job_1535984902382_0006")
val setup = job.setupProgress()
{code}
When job runnin:
{code}
scala> val setup = job.setupProgress()
setup: Float = 1.0
{code}
When job already finished:
{code}
scala> val setup = job.setupProgress()
setup: Float = 0.0
{code}
JobReport interface includes getCleanupProgress and getSetupProgress methods. But these methods ignore during creating JobReport in CompletedJob#constructJobReport(). setupProgress and cleanupProgress properties set in JobImpl class but JobFinishedEvent and JobUnsuccessfulCompletionEvent do not include this information. As the result, these properties do not include in jhist file.
We can add these properties to jhist file but of course, this is the incompatible change. These changes should be applied for a major release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org