You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "tartarus (JIRA)" <ji...@apache.org> on 2018/03/22 13:50:00 UTC
[jira] [Commented] (MAPREDUCE-7068) Fix Reduce Exception was
overwrited by ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409542#comment-16409542 ]
tartarus commented on MAPREDUCE-7068:
-------------------------------------
[~steve_l] please take a look.
> Fix Reduce Exception was overwrited by ReduceTask
> -------------------------------------------------
>
> Key: MAPREDUCE-7068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
> Affects Versions: 2.7.1
> Environment: CentOS 7
> Hadoop-2.7.1
> Hive-1.2.1
> Reporter: tartarus
> Priority: Major
> Attachments: MAPREDUCE_7068.patch
>
>
>
> {code:java}
> try {
> //increment processed counter only if skipping feature is enabled
> boolean incrProcCount = SkipBadRecords.getReducerMaxSkipGroups(job)>0 &&
> SkipBadRecords.getAutoIncrReducerProcCount(job);
>
> ReduceValuesIterator<INKEY,INVALUE> values = isSkipping() ?
> new SkippingReduceValuesIterator<INKEY,INVALUE>(rIter,
> comparator, keyClass, valueClass,
> job, reporter, umbilical) :
> new ReduceValuesIterator<INKEY,INVALUE>(rIter,
> comparator, keyClass, valueClass,
> job, reporter);
> values.informReduceProgress();
> while (values.more()) {
> reduceInputKeyCounter.increment(1);
> reducer.reduce(values.getKey(), values, collector, reporter);
> if(incrProcCount) {
> reporter.incrCounter(SkipBadRecords.COUNTER_GROUP,
> SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS, 1);
> }
> values.nextKey();
> values.informReduceProgress();
> }
> reducer.close();
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanupWithLogger(LOG, reducer);
> closeQuietly(out, reporter);
> }
> }
> {code}
> if {color:#d04437}reducer.close();{color} throw Exception , {color:#d04437}reducer = null;{color} will not run, then {color:#d04437}IOUtils.cleanupWithLogger(LOG, reducer); {color}
>
> will throw Exception and overwrite the Exception of reducer.close();
> so we should catch it and print log to help targeting issues
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org