You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Radim Kolar <hs...@filez.com> on 2012/03/24 18:30:40 UTC

Mappers only job, output sorted?

i have mappers only job - number of reducers set to 0. Its hadoop 0.22 
and output from job is this:

2012-03-24 18:24:22,117 WARN org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using 
builtin-java classes where applicable
2012-03-24 18:24:22,734 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
Initializing JVM Metrics with processName=MAP, sessionId=
2012-03-24 18:24:37,748 WARN org.apache.hadoop.conf.Configuration: 
user.name is deprecated. Instead, use mapreduce.job.user.name
2012-03-24 18:24:38,718 INFO org.apache.hadoop.mapred.Task:  Using 
ResourceCalculatorPlugin : null
2012-03-24 18:24:41,447 INFO org.apache.hadoop.mapred.Task: 
Task:attempt_201201250916_0074_m_000036_0 is done. And is in the process 
of commiting
2012-03-24 18:24:42,472 INFO org.apache.hadoop.mapred.Task: Task 
attempt_201201250916_0074_m_000036_0 is allowed to commit now
2012-03-24 18:24:42,478 INFO 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output 
of task 'attempt_201201250916_0074_m_000036_0' to /cznewgen/db-fixed
2012-03-24 18:24:42,479 INFO org.apache.hadoop.mapred.Task: Task 
'attempt_201201250916_0074_m_000036_0' done.

is this job still sorting keys in mapper output? Because task status for 
mapper says "map > sort". I do not need output sorted.

Re: Mappers only job, output sorted?

Posted by Bejoy Ks <be...@gmail.com>.

Hi Radim
        You are correct. If there is no reduce process then there won't be
any sort and shuffle phase. The output from mapper is copied directly to
hdfs itself.

Regards
Bejoy KS

2012/3/24 Radim Kolar <hs...@filez.com>

> i have mappers only job - number of reducers set to 0. Its hadoop 0.22 and
> output from job is this:
>
> 2012-03-24 18:24:22,117 WARN org.apache.hadoop.util.**NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-03-24 18:24:22,734 INFO org.apache.hadoop.metrics.jvm.**JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2012-03-24 18:24:37,748 WARN org.apache.hadoop.conf.**Configuration:
> user.name is deprecated. Instead, use mapreduce.job.user.name
> 2012-03-24 18:24:38,718 INFO org.apache.hadoop.mapred.Task:  Using
> ResourceCalculatorPlugin : null
> 2012-03-24 18:24:41,447 INFO org.apache.hadoop.mapred.Task:
> Task:attempt_201201250916_**0074_m_000036_0 is done. And is in the
> process of commiting
> 2012-03-24 18:24:42,472 INFO org.apache.hadoop.mapred.Task: Task
> attempt_201201250916_0074_m_**000036_0 is allowed to commit now
> 2012-03-24 18:24:42,478 INFO org.apache.hadoop.mapreduce.**lib.output.**FileOutputCommitter:
> Saved output of task 'attempt_201201250916_0074_m_**000036_0' to
> /cznewgen/db-fixed
> 2012-03-24 18:24:42,479 INFO org.apache.hadoop.mapred.Task: Task
> 'attempt_201201250916_0074_m_**000036_0' done.
>
> is this job still sorting keys in mapper output? Because task status for
> mapper says "map > sort". I do not need output sorted.
>