You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "David Rosenstrauch (JIRA)" <ji...@apache.org> on 2011/01/13 22:56:46 UTC
[jira] Commented: (MAPREDUCE-1853) MultipleOutputs does not cache
TaskAttemptContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981505#action_12981505 ]
David Rosenstrauch commented on MAPREDUCE-1853:
-----------------------------------------------
Is there any workaround to this issue for those of us who are still running 0.20?
I have a job that very much lends itself to using the MultipleOutputs functionality, but I can't use it if a bug like this crushes the job's performance.
Anyone have any alternate suggestions?
> MultipleOutputs does not cache TaskAttemptContext
> -------------------------------------------------
>
> Key: MAPREDUCE-1853
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1853
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0, 0.22.0
> Environment: OSX 10.6
> java6
> Reporter: Torsten Curdt
> Priority: Critical
> Fix For: 0.21.0, 0.22.0
>
> Attachments: cache-task-attempts.diff
>
>
> In MultipleOutputs there is
> [code]
> private TaskAttemptContext getContext(String nameOutput) throws IOException {
> // The following trick leverages the instantiation of a record writer via
> // the job thus supporting arbitrary output formats.
> Job job = new Job(context.getConfiguration());
> job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
> job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
> job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
> TaskAttemptContext taskContext =
> new TaskAttemptContextImpl(job.getConfiguration(),
> context.getTaskAttemptID());
> return taskContext;
> }
> [code]
> so for every reduce call it creates a new Job instance ...which creates a new LocalJobRunner.
> That does not sound like a good idea.
> You end up with a flood of "jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized"
> This should probably also be added to 0.22.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.