You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by abhiTowson cal <ab...@gmail.com> on 2012/07/29 20:30:43 UTC
Fwd:
Hi All,
I am getting problem that job is running in localrunner rather than
the cluster enviormnent.
And when am running the job i would not be able to see the job id in
the resource manager UI
Can you please go through the issues and let me know ASAP.
sudo -u hdfs hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
1000000 /benchmark/teragen/input
12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id
12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
Generating 1000000 using 1 maps with step of 1000000
12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in config null
12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapred.FileOutputCommitter
12/07/29 13:35:59 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
name and BYTES_READ as counter name instead
12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
is done. And is in the process of commiting
12/07/29 13:36:01 INFO mapred.LocalJobRunner:
12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
is allowed to commit now
12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
task 'attempt_local_0001_m_000000_0' to
hdfs://hadoop-master-1/benchmark/teragen/input
12/07/29 13:36:01 INFO mapred.LocalJobRunner:
12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes read=142686
12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
written=220956
12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read operations=0
12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
read operations=0
12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write operations=0
12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
written=100000000
12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read operations=1
12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
read operations=0
12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write operations=2
12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
usage (bytes)=124715008
12/07/29 13:36:02 INFO mapred.JobClient:
org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
Regards
Abhishek
Re:
Posted by anil gupta <an...@gmail.com>.
Seems like you are also stuck in the same problem as i am... I am going to
work on changing my conf tomorrow to fix this. How much memory your node
has?
Check the logs of Nodemanagers..At the bottom of the logs file you will see
that NM is stopping some components.(sorry i cant recall the exact name)
~Anil
On Sun, Jul 29, 2012 at 1:08 PM, abhiTowson cal
<ab...@gmail.com>wrote:
> Hi anil,
>
> Thanks for the reply.Same as your case my pi job is haulted and their
> is no progress.
>
> Regards
> Abhishek
>
> On Sun, Jul 29, 2012 at 3:31 PM, anil gupta <an...@gmail.com> wrote:
> > Hi Abhishek,
> >
> > I didnt mean to ask you whether it returns result or not. I meant that
> you
> > should check that the classpath is correct. It should have the
> directories
> > where yarn is installed.
> >
> > ~Anil
> >
> > On Sun, Jul 29, 2012 at 12:23 PM, abhiTowson cal
> > <ab...@gmail.com>wrote:
> >
> >> hi anil,
> >>
> >> Hadoop class path is also working fine.
> >>
> >> Regards
> >> Abhishek
> >>
> >> Thanks for
> >>
> >> On Sun, Jul 29, 2012 at 3:20 PM, abhiTowson cal
> >> <ab...@gmail.com> wrote:
> >> > Hi Anil,
> >> > Iam using chd4 with yarn.
> >> >
> >> > On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com>
> >> wrote:
> >> >> Are you using cdh4? In you cluster are you using yarn or mr1?
> >> >> Check the classpath of Hadoop by Hadoop classpath command.
> >> >>
> >> >> Best Regards,
> >> >> Anil
> >> >>
> >> >> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <
> abhishek.dodda1@gmail.com>
> >> wrote:
> >> >>
> >> >>> HI Anil,
> >> >>>
> >> >>> I have already tried this,but issue could not be resolved.
> >> >>>
> >> >>> Regards
> >> >>> Abhishek
> >> >>>
> >> >>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com>
> >> wrote:
> >> >>>> Hi Abhishek,
> >> >>>>
> >> >>>> Once you make sure that whatever Harsh said in the previous email
> is
> >> >>>> present in the cluster and then also the job runs in Local Mode.
> Then
> >> try
> >> >>>> running the job with hadoop --config option.
> >> >>>> Refer to this discussion for more detail:
> >> >>>>
> >>
> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
> >> >>>>
> >> >>>> HTH,
> >> >>>> Anil
> >> >>>>
> >> >>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com>
> wrote:
> >> >>>>
> >> >>>>> For a job to get submitted to a cluster, you will need proper
> client
> >> >>>>> configurations. Have you configured your mapred-site.xml and
> >> >>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
> >> >>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
> >> >>>>>
> >> >>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
> >> >>>>> <ab...@gmail.com> wrote:
> >> >>>>>> Hi All,
> >> >>>>>>
> >> >>>>>> I am getting problem that job is running in localrunner rather
> than
> >> >>>>>> the cluster enviormnent.
> >> >>>>>> And when am running the job i would not be able to see the job
> id in
> >> >>>>>> the resource manager UI
> >> >>>>>>
> >> >>>>>> Can you please go through the issues and let me know ASAP.
> >> >>>>>>
> >> >>>>>> sudo -u hdfs hadoop jar
> >> >>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> >> >>>>>> 1000000 /benchmark/teragen/input
> >> >>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is
> >> deprecated.
> >> >>>>>> Instead, use dfs.metrics.session-id
> >> >>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics
> with
> >> >>>>>> processName=JobTracker, sessionId=
> >> >>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the
> >> native-hadoop
> >> >>>>> library
> >> >>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser
> >> for
> >> >>>>>> parsing the arguments. Applications should implement Tool for the
> >> >>>>>> same.
> >> >>>>>> Generating 1000000 using 1 maps with step of 1000000
> >> >>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job:
> job_local_0001
> >> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter
> set in
> >> >>>>> config null
> >> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> >> >>>>>> org.apache.hadoop.mapred.FileOutputCommitter
> >> >>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> >> >>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> >> >>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
> >> >>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit
> >> code 0
> >> >>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using
> ResourceCalculatorPlugin
> >> :
> >> >>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> >> >>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> >> >>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as
> group
> >> >>>>>> name and BYTES_READ as counter name instead
> >> >>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> >> >>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.Task:
> >> Task:attempt_local_0001_m_000000_0
> >> >>>>>> is done. And is in the process of commiting
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> >> attempt_local_0001_m_000000_0
> >> >>>>>> is allowed to commit now
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output
> of
> >> >>>>>> task 'attempt_local_0001_m_000000_0' to
> >> >>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> >> 'attempt_local_0001_m_000000_0'
> >> >>>>> done.
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete:
> >> job_local_0001
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of
> bytes
> >> >>>>> read=142686
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of
> bytes
> >> >>>>>> written=220956
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
> >> >>>>> operations=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of
> large
> >> >>>>>> read operations=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of
> write
> >> >>>>> operations=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of
> bytes
> >> read=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of
> bytes
> >> >>>>>> written=100000000
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
> >> >>>>> operations=1
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of
> large
> >> >>>>>> read operations=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of
> write
> >> >>>>> operations=2
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input
> >> records=1000000
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output
> >> records=1000000
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent
> (ms)=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory
> (bytes)
> >> >>>>> snapshot=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory
> (bytes)
> >> >>>>> snapshot=0
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> >> >>>>>> usage (bytes)=124715008
> >> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
> >> >>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> >> >>>>>>
> >> >>>>>> Regards
> >> >>>>>> Abhishek
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Harsh J
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Thanks & Regards,
> >> >>>> Anil Gupta
> >>
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re:
Posted by abhiTowson cal <ab...@gmail.com>.
Hi anil,
Thanks for the reply.Same as your case my pi job is haulted and their
is no progress.
Regards
Abhishek
On Sun, Jul 29, 2012 at 3:31 PM, anil gupta <an...@gmail.com> wrote:
> Hi Abhishek,
>
> I didnt mean to ask you whether it returns result or not. I meant that you
> should check that the classpath is correct. It should have the directories
> where yarn is installed.
>
> ~Anil
>
> On Sun, Jul 29, 2012 at 12:23 PM, abhiTowson cal
> <ab...@gmail.com>wrote:
>
>> hi anil,
>>
>> Hadoop class path is also working fine.
>>
>> Regards
>> Abhishek
>>
>> Thanks for
>>
>> On Sun, Jul 29, 2012 at 3:20 PM, abhiTowson cal
>> <ab...@gmail.com> wrote:
>> > Hi Anil,
>> > Iam using chd4 with yarn.
>> >
>> > On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com>
>> wrote:
>> >> Are you using cdh4? In you cluster are you using yarn or mr1?
>> >> Check the classpath of Hadoop by Hadoop classpath command.
>> >>
>> >> Best Regards,
>> >> Anil
>> >>
>> >> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com>
>> wrote:
>> >>
>> >>> HI Anil,
>> >>>
>> >>> I have already tried this,but issue could not be resolved.
>> >>>
>> >>> Regards
>> >>> Abhishek
>> >>>
>> >>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com>
>> wrote:
>> >>>> Hi Abhishek,
>> >>>>
>> >>>> Once you make sure that whatever Harsh said in the previous email is
>> >>>> present in the cluster and then also the job runs in Local Mode. Then
>> try
>> >>>> running the job with hadoop --config option.
>> >>>> Refer to this discussion for more detail:
>> >>>>
>> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>> >>>>
>> >>>> HTH,
>> >>>> Anil
>> >>>>
>> >>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>
>> >>>>> For a job to get submitted to a cluster, you will need proper client
>> >>>>> configurations. Have you configured your mapred-site.xml and
>> >>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>> >>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
>> >>>>>
>> >>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>> >>>>> <ab...@gmail.com> wrote:
>> >>>>>> Hi All,
>> >>>>>>
>> >>>>>> I am getting problem that job is running in localrunner rather than
>> >>>>>> the cluster enviormnent.
>> >>>>>> And when am running the job i would not be able to see the job id in
>> >>>>>> the resource manager UI
>> >>>>>>
>> >>>>>> Can you please go through the issues and let me know ASAP.
>> >>>>>>
>> >>>>>> sudo -u hdfs hadoop jar
>> >>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>> >>>>>> 1000000 /benchmark/teragen/input
>> >>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is
>> deprecated.
>> >>>>>> Instead, use dfs.metrics.session-id
>> >>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> >>>>>> processName=JobTracker, sessionId=
>> >>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the
>> native-hadoop
>> >>>>> library
>> >>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser
>> for
>> >>>>>> parsing the arguments. Applications should implement Tool for the
>> >>>>>> same.
>> >>>>>> Generating 1000000 using 1 maps with step of 1000000
>> >>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>> >>>>> config null
>> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>> >>>>>> org.apache.hadoop.mapred.FileOutputCommitter
>> >>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>> >>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>> >>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>> >>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit
>> code 0
>> >>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin
>> :
>> >>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>> >>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>> >>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>> >>>>>> name and BYTES_READ as counter name instead
>> >>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>> >>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>> >>>>>> 12/07/29 13:36:01 INFO mapred.Task:
>> Task:attempt_local_0001_m_000000_0
>> >>>>>> is done. And is in the process of commiting
>> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
>> attempt_local_0001_m_000000_0
>> >>>>>> is allowed to commit now
>> >>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>> >>>>>> task 'attempt_local_0001_m_000000_0' to
>> >>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
>> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
>> 'attempt_local_0001_m_000000_0'
>> >>>>> done.
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete:
>> job_local_0001
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>> >>>>> read=142686
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>> >>>>>> written=220956
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>> >>>>> operations=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>> >>>>>> read operations=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>> >>>>> operations=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>> read=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>> >>>>>> written=100000000
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>> >>>>> operations=1
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>> >>>>>> read operations=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>> >>>>> operations=2
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input
>> records=1000000
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output
>> records=1000000
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>> >>>>> snapshot=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>> >>>>> snapshot=0
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>> >>>>>> usage (bytes)=124715008
>> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
>> >>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>> >>>>>>
>> >>>>>> Regards
>> >>>>>> Abhishek
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Harsh J
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Thanks & Regards,
>> >>>> Anil Gupta
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re:
Posted by anil gupta <an...@gmail.com>.
Hi Abhishek,
I didnt mean to ask you whether it returns result or not. I meant that you
should check that the classpath is correct. It should have the directories
where yarn is installed.
~Anil
On Sun, Jul 29, 2012 at 12:23 PM, abhiTowson cal
<ab...@gmail.com>wrote:
> hi anil,
>
> Hadoop class path is also working fine.
>
> Regards
> Abhishek
>
> Thanks for
>
> On Sun, Jul 29, 2012 at 3:20 PM, abhiTowson cal
> <ab...@gmail.com> wrote:
> > Hi Anil,
> > Iam using chd4 with yarn.
> >
> > On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com>
> wrote:
> >> Are you using cdh4? In you cluster are you using yarn or mr1?
> >> Check the classpath of Hadoop by Hadoop classpath command.
> >>
> >> Best Regards,
> >> Anil
> >>
> >> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com>
> wrote:
> >>
> >>> HI Anil,
> >>>
> >>> I have already tried this,but issue could not be resolved.
> >>>
> >>> Regards
> >>> Abhishek
> >>>
> >>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com>
> wrote:
> >>>> Hi Abhishek,
> >>>>
> >>>> Once you make sure that whatever Harsh said in the previous email is
> >>>> present in the cluster and then also the job runs in Local Mode. Then
> try
> >>>> running the job with hadoop --config option.
> >>>> Refer to this discussion for more detail:
> >>>>
> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
> >>>>
> >>>> HTH,
> >>>> Anil
> >>>>
> >>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>>> For a job to get submitted to a cluster, you will need proper client
> >>>>> configurations. Have you configured your mapred-site.xml and
> >>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
> >>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
> >>>>>
> >>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
> >>>>> <ab...@gmail.com> wrote:
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I am getting problem that job is running in localrunner rather than
> >>>>>> the cluster enviormnent.
> >>>>>> And when am running the job i would not be able to see the job id in
> >>>>>> the resource manager UI
> >>>>>>
> >>>>>> Can you please go through the issues and let me know ASAP.
> >>>>>>
> >>>>>> sudo -u hdfs hadoop jar
> >>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> >>>>>> 1000000 /benchmark/teragen/input
> >>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is
> deprecated.
> >>>>>> Instead, use dfs.metrics.session-id
> >>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> >>>>>> processName=JobTracker, sessionId=
> >>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the
> native-hadoop
> >>>>> library
> >>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser
> for
> >>>>>> parsing the arguments. Applications should implement Tool for the
> >>>>>> same.
> >>>>>> Generating 1000000 using 1 maps with step of 1000000
> >>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
> >>>>> config null
> >>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> >>>>>> org.apache.hadoop.mapred.FileOutputCommitter
> >>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> >>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> >>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
> >>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit
> code 0
> >>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin
> :
> >>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> >>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> >>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> >>>>>> name and BYTES_READ as counter name instead
> >>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> >>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> >>>>>> 12/07/29 13:36:01 INFO mapred.Task:
> Task:attempt_local_0001_m_000000_0
> >>>>>> is done. And is in the process of commiting
> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> attempt_local_0001_m_000000_0
> >>>>>> is allowed to commit now
> >>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> >>>>>> task 'attempt_local_0001_m_000000_0' to
> >>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
> >>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> 'attempt_local_0001_m_000000_0'
> >>>>> done.
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete:
> job_local_0001
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> >>>>> read=142686
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> >>>>>> written=220956
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
> >>>>> operations=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
> >>>>>> read operations=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
> >>>>> operations=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> read=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> >>>>>> written=100000000
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
> >>>>> operations=1
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
> >>>>>> read operations=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
> >>>>> operations=2
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input
> records=1000000
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output
> records=1000000
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
> >>>>> snapshot=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
> >>>>> snapshot=0
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> >>>>>> usage (bytes)=124715008
> >>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
> >>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> >>>>>>
> >>>>>> Regards
> >>>>>> Abhishek
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks & Regards,
> >>>> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re:
Posted by abhiTowson cal <ab...@gmail.com>.
hi anil,
Hadoop class path is also working fine.
Regards
Abhishek
Thanks for
On Sun, Jul 29, 2012 at 3:20 PM, abhiTowson cal
<ab...@gmail.com> wrote:
> Hi Anil,
> Iam using chd4 with yarn.
>
> On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com> wrote:
>> Are you using cdh4? In you cluster are you using yarn or mr1?
>> Check the classpath of Hadoop by Hadoop classpath command.
>>
>> Best Regards,
>> Anil
>>
>> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com> wrote:
>>
>>> HI Anil,
>>>
>>> I have already tried this,but issue could not be resolved.
>>>
>>> Regards
>>> Abhishek
>>>
>>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com> wrote:
>>>> Hi Abhishek,
>>>>
>>>> Once you make sure that whatever Harsh said in the previous email is
>>>> present in the cluster and then also the job runs in Local Mode. Then try
>>>> running the job with hadoop --config option.
>>>> Refer to this discussion for more detail:
>>>> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>>>>
>>>> HTH,
>>>> Anil
>>>>
>>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>>> For a job to get submitted to a cluster, you will need proper client
>>>>> configurations. Have you configured your mapred-site.xml and
>>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
>>>>>
>>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>>>>> <ab...@gmail.com> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I am getting problem that job is running in localrunner rather than
>>>>>> the cluster enviormnent.
>>>>>> And when am running the job i would not be able to see the job id in
>>>>>> the resource manager UI
>>>>>>
>>>>>> Can you please go through the issues and let me know ASAP.
>>>>>>
>>>>>> sudo -u hdfs hadoop jar
>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>>>>>> 1000000 /benchmark/teragen/input
>>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
>>>>>> Instead, use dfs.metrics.session-id
>>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>>>>>> processName=JobTracker, sessionId=
>>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>>>> library
>>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>>> parsing the arguments. Applications should implement Tool for the
>>>>>> same.
>>>>>> Generating 1000000 using 1 maps with step of 1000000
>>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>>>>> config null
>>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>>>>>> org.apache.hadoop.mapred.FileOutputCommitter
>>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
>>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
>>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>>>>>> name and BYTES_READ as counter name instead
>>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
>>>>>> is done. And is in the process of commiting
>>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
>>>>>> is allowed to commit now
>>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>>>>>> task 'attempt_local_0001_m_000000_0' to
>>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
>>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
>>>>> done.
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>>> read=142686
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>>>> written=220956
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>>>>> operations=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>>>>>> read operations=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>>>>> operations=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>>>>>> written=100000000
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>>>>> operations=1
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>>>>>> read operations=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>>>>> operations=2
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>>>>> snapshot=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>>>>> snapshot=0
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>>>>>> usage (bytes)=124715008
>>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
>>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>>>>>>
>>>>>> Regards
>>>>>> Abhishek
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anil Gupta
Re:
Posted by anil gupta <an...@gmail.com>.
Hi Abhishek,
I faced similar problem with cdh4 few days ago. THe problem i found out was
the classpath. The user from which i installed the cdh packages was having
the right classpath but the users "yarn" and "hdfs" were having the
incorrect classpath. When i was trying to run the job as yarn or hdfs user,
it used to run in local mode and when i ran it with "root"(root user was
used to install cdh4) it ran in distributed mode.
Can you try running the job with user you used to install cdh4.
Also compare the classpath of "hdfs" and "yarn" and user used while
installation by running the following:
sudo -u hdfs hadoop classpath
sudo -u yarn hadoop classpath
sudo -u $installation_user classpath
~Anil
On Sun, Jul 29, 2012 at 12:20 PM, abhiTowson cal
<ab...@gmail.com>wrote:
> Hi Anil,
> Iam using chd4 with yarn.
>
> On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com> wrote:
> > Are you using cdh4? In you cluster are you using yarn or mr1?
> > Check the classpath of Hadoop by Hadoop classpath command.
> >
> > Best Regards,
> > Anil
> >
> > On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com>
> wrote:
> >
> >> HI Anil,
> >>
> >> I have already tried this,but issue could not be resolved.
> >>
> >> Regards
> >> Abhishek
> >>
> >> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com>
> wrote:
> >>> Hi Abhishek,
> >>>
> >>> Once you make sure that whatever Harsh said in the previous email is
> >>> present in the cluster and then also the job runs in Local Mode. Then
> try
> >>> running the job with hadoop --config option.
> >>> Refer to this discussion for more detail:
> >>>
> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
> >>>
> >>> HTH,
> >>> Anil
> >>>
> >>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>
> >>>> For a job to get submitted to a cluster, you will need proper client
> >>>> configurations. Have you configured your mapred-site.xml and
> >>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
> >>>> /etc/hadoop/conf/yarn-site.xml at the client node?
> >>>>
> >>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
> >>>> <ab...@gmail.com> wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> I am getting problem that job is running in localrunner rather than
> >>>>> the cluster enviormnent.
> >>>>> And when am running the job i would not be able to see the job id in
> >>>>> the resource manager UI
> >>>>>
> >>>>> Can you please go through the issues and let me know ASAP.
> >>>>>
> >>>>> sudo -u hdfs hadoop jar
> >>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> >>>>> 1000000 /benchmark/teragen/input
> >>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
> >>>>> Instead, use dfs.metrics.session-id
> >>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> >>>>> processName=JobTracker, sessionId=
> >>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the
> native-hadoop
> >>>> library
> >>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
> >>>>> parsing the arguments. Applications should implement Tool for the
> >>>>> same.
> >>>>> Generating 1000000 using 1 maps with step of 1000000
> >>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> >>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
> >>>> config null
> >>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> >>>>> org.apache.hadoop.mapred.FileOutputCommitter
> >>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> >>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> >>>>> org.apache.hadoop.mapreduce.TaskCounter instead
> >>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit
> code 0
> >>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
> >>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> >>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> >>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> >>>>> name and BYTES_READ as counter name instead
> >>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> >>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> >>>>> 12/07/29 13:36:01 INFO mapred.Task:
> Task:attempt_local_0001_m_000000_0
> >>>>> is done. And is in the process of commiting
> >>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> attempt_local_0001_m_000000_0
> >>>>> is allowed to commit now
> >>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> >>>>> task 'attempt_local_0001_m_000000_0' to
> >>>>> hdfs://hadoop-master-1/benchmark/teragen/input
> >>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> >>>>> 12/07/29 13:36:01 INFO mapred.Task: Task
> 'attempt_local_0001_m_000000_0'
> >>>> done.
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> >>>> read=142686
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> >>>>> written=220956
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
> >>>> operations=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
> >>>>> read operations=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
> >>>> operations=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> read=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> >>>>> written=100000000
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
> >>>> operations=1
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
> >>>>> read operations=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
> >>>> operations=2
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input
> records=1000000
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output
> records=1000000
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
> >>>> snapshot=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
> >>>> snapshot=0
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> >>>>> usage (bytes)=124715008
> >>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
> >>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> >>>>>
> >>>>> Regards
> >>>>> Abhishek
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Harsh J
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re:
Posted by abhiTowson cal <ab...@gmail.com>.
Hi Anil,
Iam using chd4 with yarn.
On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <an...@gmail.com> wrote:
> Are you using cdh4? In you cluster are you using yarn or mr1?
> Check the classpath of Hadoop by Hadoop classpath command.
>
> Best Regards,
> Anil
>
> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com> wrote:
>
>> HI Anil,
>>
>> I have already tried this,but issue could not be resolved.
>>
>> Regards
>> Abhishek
>>
>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com> wrote:
>>> Hi Abhishek,
>>>
>>> Once you make sure that whatever Harsh said in the previous email is
>>> present in the cluster and then also the job runs in Local Mode. Then try
>>> running the job with hadoop --config option.
>>> Refer to this discussion for more detail:
>>> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>>>
>>> HTH,
>>> Anil
>>>
>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> For a job to get submitted to a cluster, you will need proper client
>>>> configurations. Have you configured your mapred-site.xml and
>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
>>>>
>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>>>> <ab...@gmail.com> wrote:
>>>>> Hi All,
>>>>>
>>>>> I am getting problem that job is running in localrunner rather than
>>>>> the cluster enviormnent.
>>>>> And when am running the job i would not be able to see the job id in
>>>>> the resource manager UI
>>>>>
>>>>> Can you please go through the issues and let me know ASAP.
>>>>>
>>>>> sudo -u hdfs hadoop jar
>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>>>>> 1000000 /benchmark/teragen/input
>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
>>>>> Instead, use dfs.metrics.session-id
>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>>>>> processName=JobTracker, sessionId=
>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>>> library
>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the
>>>>> same.
>>>>> Generating 1000000 using 1 maps with step of 1000000
>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>>>> config null
>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>>>>> org.apache.hadoop.mapred.FileOutputCommitter
>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>>>>> name and BYTES_READ as counter name instead
>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
>>>>> is done. And is in the process of commiting
>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
>>>>> is allowed to commit now
>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>>>>> task 'attempt_local_0001_m_000000_0' to
>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
>>>> done.
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>> read=142686
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>>> written=220956
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>>>> operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>>>>> read operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>>>> operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>>>>> written=100000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>>>> operations=1
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>>>>> read operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>>>> operations=2
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>>>> snapshot=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>>>> snapshot=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>>>>> usage (bytes)=124715008
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>>>>>
>>>>> Regards
>>>>> Abhishek
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
Re:
Posted by Anil Gupta <an...@gmail.com>.
Are you using cdh4? In you cluster are you using yarn or mr1?
Check the classpath of Hadoop by Hadoop classpath command.
Best Regards,
Anil
On Jul 29, 2012, at 12:12 PM, abhiTowson cal <ab...@gmail.com> wrote:
> HI Anil,
>
> I have already tried this,but issue could not be resolved.
>
> Regards
> Abhishek
>
> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com> wrote:
>> Hi Abhishek,
>>
>> Once you make sure that whatever Harsh said in the previous email is
>> present in the cluster and then also the job runs in Local Mode. Then try
>> running the job with hadoop --config option.
>> Refer to this discussion for more detail:
>> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>>
>> HTH,
>> Anil
>>
>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> For a job to get submitted to a cluster, you will need proper client
>>> configurations. Have you configured your mapred-site.xml and
>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>>> /etc/hadoop/conf/yarn-site.xml at the client node?
>>>
>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>>> <ab...@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>> I am getting problem that job is running in localrunner rather than
>>>> the cluster enviormnent.
>>>> And when am running the job i would not be able to see the job id in
>>>> the resource manager UI
>>>>
>>>> Can you please go through the issues and let me know ASAP.
>>>>
>>>> sudo -u hdfs hadoop jar
>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>>>> 1000000 /benchmark/teragen/input
>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
>>>> Instead, use dfs.metrics.session-id
>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>>>> processName=JobTracker, sessionId=
>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
>>>> parsing the arguments. Applications should implement Tool for the
>>>> same.
>>>> Generating 1000000 using 1 maps with step of 1000000
>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>>> config null
>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>>>> org.apache.hadoop.mapred.FileOutputCommitter
>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>>>> name and BYTES_READ as counter name instead
>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>>>> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
>>>> is done. And is in the process of commiting
>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
>>>> is allowed to commit now
>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>>>> task 'attempt_local_0001_m_000000_0' to
>>>> hdfs://hadoop-master-1/benchmark/teragen/input
>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
>>> done.
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>> read=142686
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>> written=220956
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>>> operations=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>>>> read operations=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>>> operations=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>>>> written=100000000
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>>> operations=1
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>>>> read operations=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>>> operations=2
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>>> snapshot=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>>> snapshot=0
>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>>>> usage (bytes)=124715008
>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>>>>
>>>> Regards
>>>> Abhishek
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
Re:
Posted by abhiTowson cal <ab...@gmail.com>.
HI Anil,
I have already tried this,but issue could not be resolved.
Regards
Abhishek
On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <an...@gmail.com> wrote:
> Hi Abhishek,
>
> Once you make sure that whatever Harsh said in the previous email is
> present in the cluster and then also the job runs in Local Mode. Then try
> running the job with hadoop --config option.
> Refer to this discussion for more detail:
> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>
> HTH,
> Anil
>
> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> For a job to get submitted to a cluster, you will need proper client
>> configurations. Have you configured your mapred-site.xml and
>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>> /etc/hadoop/conf/yarn-site.xml at the client node?
>>
>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>> <ab...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am getting problem that job is running in localrunner rather than
>> > the cluster enviormnent.
>> > And when am running the job i would not be able to see the job id in
>> > the resource manager UI
>> >
>> > Can you please go through the issues and let me know ASAP.
>> >
>> > sudo -u hdfs hadoop jar
>> > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>> > 1000000 /benchmark/teragen/input
>> > 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
>> > Instead, use dfs.metrics.session-id
>> > 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> > processName=JobTracker, sessionId=
>> > 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> > 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
>> > parsing the arguments. Applications should implement Tool for the
>> > same.
>> > Generating 1000000 using 1 maps with step of 1000000
>> > 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>> > 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>> config null
>> > 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>> > org.apache.hadoop.mapred.FileOutputCommitter
>> > 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>> > org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>> > org.apache.hadoop.mapreduce.TaskCounter instead
>> > 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
>> > 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
>> > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>> > 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>> > MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>> > name and BYTES_READ as counter name instead
>> > 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>> > 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>> > 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
>> > is done. And is in the process of commiting
>> > 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>> > 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
>> > is allowed to commit now
>> > 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>> > task 'attempt_local_0001_m_000000_0' to
>> > hdfs://hadoop-master-1/benchmark/teragen/input
>> > 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>> > 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
>> done.
>> > 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>> > 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>> read=142686
>> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>> > written=220956
>> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>> operations=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>> > read operations=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>> operations=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>> > written=100000000
>> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>> operations=1
>> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>> > read operations=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>> operations=2
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=0
>> > 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>> > usage (bytes)=124715008
>> > 12/07/29 13:36:02 INFO mapred.JobClient:
>> > org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>> >
>> > Regards
>> > Abhishek
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re:
Posted by anil gupta <an...@gmail.com>.
Hi Abhishek,
Once you make sure that whatever Harsh said in the previous email is
present in the cluster and then also the job runs in Local Mode. Then try
running the job with hadoop --config option.
Refer to this discussion for more detail:
https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
HTH,
Anil
On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
> For a job to get submitted to a cluster, you will need proper client
> configurations. Have you configured your mapred-site.xml and
> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
> /etc/hadoop/conf/yarn-site.xml at the client node?
>
> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
> <ab...@gmail.com> wrote:
> > Hi All,
> >
> > I am getting problem that job is running in localrunner rather than
> > the cluster enviormnent.
> > And when am running the job i would not be able to see the job id in
> > the resource manager UI
> >
> > Can you please go through the issues and let me know ASAP.
> >
> > sudo -u hdfs hadoop jar
> > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> > 1000000 /benchmark/teragen/input
> > 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
> > Instead, use dfs.metrics.session-id
> > 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> > processName=JobTracker, sessionId=
> > 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> > 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the
> > same.
> > Generating 1000000 using 1 maps with step of 1000000
> > 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> > 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> > 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> > org.apache.hadoop.mapred.FileOutputCommitter
> > 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> > org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> > org.apache.hadoop.mapreduce.TaskCounter instead
> > 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
> > 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
> > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> > 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> > MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> > name and BYTES_READ as counter name instead
> > 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> > 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> > 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
> > is done. And is in the process of commiting
> > 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> > 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
> > is allowed to commit now
> > 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> > task 'attempt_local_0001_m_000000_0' to
> > hdfs://hadoop-master-1/benchmark/teragen/input
> > 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> > 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
> done.
> > 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> > 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
> > 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> > 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> read=142686
> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> > written=220956
> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
> operations=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
> > read operations=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
> operations=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> > written=100000000
> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
> operations=1
> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
> > read operations=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
> operations=2
> > 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> > 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
> > 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
> > 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> > 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=0
> > 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> > usage (bytes)=124715008
> > 12/07/29 13:36:02 INFO mapred.JobClient:
> > org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> >
> > Regards
> > Abhishek
>
>
>
> --
> Harsh J
>
--
Thanks & Regards,
Anil Gupta
Re:
Posted by Harsh J <ha...@cloudera.com>.
For a job to get submitted to a cluster, you will need proper client
configurations. Have you configured your mapred-site.xml and
yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
/etc/hadoop/conf/yarn-site.xml at the client node?
On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
<ab...@gmail.com> wrote:
> Hi All,
>
> I am getting problem that job is running in localrunner rather than
> the cluster enviormnent.
> And when am running the job i would not be able to see the job id in
> the resource manager UI
>
> Can you please go through the issues and let me know ASAP.
>
> sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> 1000000 /benchmark/teragen/input
> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> Generating 1000000 using 1 maps with step of 1000000
> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapred.FileOutputCommitter
> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> name and BYTES_READ as counter name instead
> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
> is done. And is in the process of commiting
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
> is allowed to commit now
> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> task 'attempt_local_0001_m_000000_0' to
> hdfs://hadoop-master-1/benchmark/teragen/input
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes read=142686
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> written=220956
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> written=100000000
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read operations=1
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write operations=2
> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> usage (bytes)=124715008
> 12/07/29 13:36:02 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>
> Regards
> Abhishek
--
Harsh J
Re:
Posted by abhiTowson cal <ab...@gmail.com>.
Hi,
Thanks for reply Harsh, These are my configuration properties
//mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-master-2:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-master-2:19888</value>
</property>
</configuration>
//Yarn-site-xml
<configuration>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop-master-2:8040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop-master-2:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop-master-2:8141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop-master-2:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/srv/storage/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/srv/storage/yarn/log</value>
</property>
</configuration>
On Sun, Jul 29, 2012 at 2:30 PM, abhiTowson cal
<ab...@gmail.com> wrote:
> Hi All,
>
> I am getting problem that job is running in localrunner rather than
> the cluster enviormnent.
> And when am running the job i would not be able to see the job id in
> the resource manager UI
>
> Can you please go through the issues and let me know ASAP.
>
> sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> 1000000 /benchmark/teragen/input
> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> Generating 1000000 using 1 maps with step of 1000000
> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapred.FileOutputCommitter
> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> name and BYTES_READ as counter name instead
> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
> is done. And is in the process of commiting
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
> is allowed to commit now
> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> task 'attempt_local_0001_m_000000_0' to
> hdfs://hadoop-master-1/benchmark/teragen/input
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes read=142686
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
> written=220956
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
> written=100000000
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read operations=1
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write operations=2
> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
> usage (bytes)=124715008
> 12/07/29 13:36:02 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>
> Regards
> Abhishek