Re: Hadoop 2.2.0 : Job not showing on resource management web UI

[Zhijie Shen <zs...@hortonworks.com>]

You were still running the local MR job, which is why you didn't see the
job on RM web UI:

<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>

This config is not in bad format. Please change to "*name*->
mapreduce.framework.name and *value*->yarn".


On Sat, Jun 28, 2014 at 8:54 PM, Shing Hing Man <ma...@yahoo.com> wrote:

> Hi,
>  I have set up a single node Hadoop 2.2.0 .
>  I can submit wordcount hadoop job to YARN from the command line.
>  It run successfully and I checked the result is in hdfs.
>   matmsh@gauss:~> hadoop jar
> ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount input output6
> 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
> 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> 14/06/28 13:40:00 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class
> is deprecated. Instead, use mapreduce.job.combine.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class
> is deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local28288787_0001
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task:
> attempt_local28288787_0001_m_000000_0
> 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree :
> [ ]
> 14/06/28 13:40:01 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
> 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> .....
>
> However, the above job does not show up at the web UI :
> http://localhost:8088/cluster/apps
>
> My configurations are below,
> mapred-site.xml :
> <configuration>
>
>  <property>
>  <value>mapreduce.framework.name</value>
>  <name>yarn</name>
>  </property>
>
> </configuration>
>
> -------
> hdfs-site.xml
>
>
> <configuration>
>  <property>
>  <name>dfs.replication</name>
>  <value>1</value>
>  </property>
>
>  <property>
>  <name>dfs.namenode.name.dir</name>
>  <value>file:/home/matmsh/temp/hadoop2/namenode</value>
>  <description>The name of the default file system.</description>
>  </property>
>
>  <property>
>  <name>dfs.datanode.data.dir</name>
>  <value>file:/home/matmsh/temp/hadoop2/datanode</value>
>  <description>The name of the default file system.</description>
>  </property>
>
> </configuration>
> ----------------
> yarn-site.xml
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>
>  <property>
>  <name>yarn.nodemanager.aux-services</name>
>  <value>mapreduce_shuffle</value>
>  </property>
>   <property>
>  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>  </property>
>
>
> </configuration>
>
> ------------------------------------
>
> What other configurations are needed to show the job in the web UI ?
> Thanks in advance for any assistance !
>
> Shing
>
>
>


-- 
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.