You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/11/26 06:27:23 UTC
why my terasort job become a local job?
hi,maillist:
i run terasort in my hadoop cluster,and it run as a local job,i
do not know why ,anyone can help?
i use hadoop version is CDH4.4
# sudo -u hdfs hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
teragen 10000000 /alex/terasort/10G-input
......
13/11/26 11:57:39 INFO mapred.Task:
Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
of commiting
13/11/26 11:57:39 INFO mapred.LocalJobRunner:
13/11/26 11:57:39 INFO mapred.Task: Task
attempt_local321416814_0001_m_000000_0 is allowed to commit now
13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
'attempt_local321416814_0001_m_000000_0' to
hdfs://product/alex/terasort/10G-input
13/11/26 11:57:39 INFO mapred.LocalJobRunner:
13/11/26 11:57:39 INFO mapred.Task: Task
'attempt_local321416814_0001_m_000000_0' done.
13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
attempt_local321416814_0001_m_000000_0
13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
job_local321416814_0001
13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
Re: why my terasort job become a local job?
Posted by ch huang <ju...@gmail.com>.
yes ,i did
# grep -C 3 framework /etc/hadoop/conf/mapred-site.xml
<configuration>
<!-- YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:36 PM, Jeff Zhang <je...@gopivotal.com> wrote:
> Do you set to use yarn framework in mapred-site.xml as following ?
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
>
> On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i run terasort in my hadoop cluster,and it run as a local job,i
>> do not know why ,anyone can help?
>>
>> i use hadoop version is CDH4.4
>>
>> # sudo -u hdfs hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
>> teragen 10000000 /alex/terasort/10G-input
>> ......
>>
>> 13/11/26 11:57:39 INFO mapred.Task:
>> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
>> of commiting
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> attempt_local321416814_0001_m_000000_0 is allowed to commit now
>> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
>> 'attempt_local321416814_0001_m_000000_0' to
>> hdfs://product/alex/terasort/10G-input
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> 'attempt_local321416814_0001_m_000000_0' done.
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
>> attempt_local321416814_0001_m_000000_0
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
>> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
>> job_local321416814_0001
>> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>>
>
>
Re: why my terasort job become a local job?
Posted by ch huang <ju...@gmail.com>.
yes ,i did
# grep -C 3 framework /etc/hadoop/conf/mapred-site.xml
<configuration>
<!-- YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:36 PM, Jeff Zhang <je...@gopivotal.com> wrote:
> Do you set to use yarn framework in mapred-site.xml as following ?
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
>
> On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i run terasort in my hadoop cluster,and it run as a local job,i
>> do not know why ,anyone can help?
>>
>> i use hadoop version is CDH4.4
>>
>> # sudo -u hdfs hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
>> teragen 10000000 /alex/terasort/10G-input
>> ......
>>
>> 13/11/26 11:57:39 INFO mapred.Task:
>> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
>> of commiting
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> attempt_local321416814_0001_m_000000_0 is allowed to commit now
>> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
>> 'attempt_local321416814_0001_m_000000_0' to
>> hdfs://product/alex/terasort/10G-input
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> 'attempt_local321416814_0001_m_000000_0' done.
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
>> attempt_local321416814_0001_m_000000_0
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
>> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
>> job_local321416814_0001
>> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>>
>
>
Re: why my terasort job become a local job?
Posted by ch huang <ju...@gmail.com>.
yes ,i did
# grep -C 3 framework /etc/hadoop/conf/mapred-site.xml
<configuration>
<!-- YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:36 PM, Jeff Zhang <je...@gopivotal.com> wrote:
> Do you set to use yarn framework in mapred-site.xml as following ?
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
>
> On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i run terasort in my hadoop cluster,and it run as a local job,i
>> do not know why ,anyone can help?
>>
>> i use hadoop version is CDH4.4
>>
>> # sudo -u hdfs hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
>> teragen 10000000 /alex/terasort/10G-input
>> ......
>>
>> 13/11/26 11:57:39 INFO mapred.Task:
>> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
>> of commiting
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> attempt_local321416814_0001_m_000000_0 is allowed to commit now
>> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
>> 'attempt_local321416814_0001_m_000000_0' to
>> hdfs://product/alex/terasort/10G-input
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> 'attempt_local321416814_0001_m_000000_0' done.
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
>> attempt_local321416814_0001_m_000000_0
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
>> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
>> job_local321416814_0001
>> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>>
>
>
Re: why my terasort job become a local job?
Posted by ch huang <ju...@gmail.com>.
yes ,i did
# grep -C 3 framework /etc/hadoop/conf/mapred-site.xml
<configuration>
<!-- YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:36 PM, Jeff Zhang <je...@gopivotal.com> wrote:
> Do you set to use yarn framework in mapred-site.xml as following ?
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
>
>
> On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i run terasort in my hadoop cluster,and it run as a local job,i
>> do not know why ,anyone can help?
>>
>> i use hadoop version is CDH4.4
>>
>> # sudo -u hdfs hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
>> teragen 10000000 /alex/terasort/10G-input
>> ......
>>
>> 13/11/26 11:57:39 INFO mapred.Task:
>> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
>> of commiting
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> attempt_local321416814_0001_m_000000_0 is allowed to commit now
>> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
>> 'attempt_local321416814_0001_m_000000_0' to
>> hdfs://product/alex/terasort/10G-input
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
>> 13/11/26 11:57:39 INFO mapred.Task: Task
>> 'attempt_local321416814_0001_m_000000_0' done.
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
>> attempt_local321416814_0001_m_000000_0
>> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
>> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
>> job_local321416814_0001
>> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>>
>
>
Re: why my terasort job become a local job?
Posted by Jeff Zhang <je...@gopivotal.com>.
Do you set to use yarn framework in mapred-site.xml as following ?
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i run terasort in my hadoop cluster,and it run as a local job,i
> do not know why ,anyone can help?
>
> i use hadoop version is CDH4.4
>
> # sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
> teragen 10000000 /alex/terasort/10G-input
> ......
>
> 13/11/26 11:57:39 INFO mapred.Task:
> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
> of commiting
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> attempt_local321416814_0001_m_000000_0 is allowed to commit now
> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
> 'attempt_local321416814_0001_m_000000_0' to
> hdfs://product/alex/terasort/10G-input
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> 'attempt_local321416814_0001_m_000000_0' done.
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local321416814_0001_m_000000_0
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
> job_local321416814_0001
> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>
Re: why my terasort job become a local job?
Posted by Jeff Zhang <je...@gopivotal.com>.
Do you set to use yarn framework in mapred-site.xml as following ?
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i run terasort in my hadoop cluster,and it run as a local job,i
> do not know why ,anyone can help?
>
> i use hadoop version is CDH4.4
>
> # sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
> teragen 10000000 /alex/terasort/10G-input
> ......
>
> 13/11/26 11:57:39 INFO mapred.Task:
> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
> of commiting
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> attempt_local321416814_0001_m_000000_0 is allowed to commit now
> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
> 'attempt_local321416814_0001_m_000000_0' to
> hdfs://product/alex/terasort/10G-input
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> 'attempt_local321416814_0001_m_000000_0' done.
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local321416814_0001_m_000000_0
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
> job_local321416814_0001
> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>
Re: why my terasort job become a local job?
Posted by Jeff Zhang <je...@gopivotal.com>.
Do you set to use yarn framework in mapred-site.xml as following ?
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i run terasort in my hadoop cluster,and it run as a local job,i
> do not know why ,anyone can help?
>
> i use hadoop version is CDH4.4
>
> # sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
> teragen 10000000 /alex/terasort/10G-input
> ......
>
> 13/11/26 11:57:39 INFO mapred.Task:
> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
> of commiting
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> attempt_local321416814_0001_m_000000_0 is allowed to commit now
> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
> 'attempt_local321416814_0001_m_000000_0' to
> hdfs://product/alex/terasort/10G-input
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> 'attempt_local321416814_0001_m_000000_0' done.
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local321416814_0001_m_000000_0
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
> job_local321416814_0001
> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>
Re: why my terasort job become a local job?
Posted by Jeff Zhang <je...@gopivotal.com>.
Do you set to use yarn framework in mapred-site.xml as following ?
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
On Tue, Nov 26, 2013 at 1:27 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i run terasort in my hadoop cluster,and it run as a local job,i
> do not know why ,anyone can help?
>
> i use hadoop version is CDH4.4
>
> # sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
> teragen 10000000 /alex/terasort/10G-input
> ......
>
> 13/11/26 11:57:39 INFO mapred.Task:
> Task:attempt_local321416814_0001_m_000000_0 is done. And is in the process
> of commiting
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> attempt_local321416814_0001_m_000000_0 is allowed to commit now
> 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task
> 'attempt_local321416814_0001_m_000000_0' to
> hdfs://product/alex/terasort/10G-input
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner:
> 13/11/26 11:57:39 INFO mapred.Task: Task
> 'attempt_local321416814_0001_m_000000_0' done.
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local321416814_0001_m_000000_0
> 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0%
> 13/11/26 11:57:40 INFO mapred.JobClient: Job complete:
> job_local321416814_0001
> 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19
>