You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shing Hing Man <ma...@yahoo.com> on 2014/06/28 14:54:31 UTC
Hadoop 2.2.0 : Job not showing on resource management web UI
Hi,
I have set up a single node Hadoop 2.2.0 .
I can submit wordcount hadoop job to YARN from the command line.
It run successfully and I checked the result is in hdfs.
matmsh@gauss:~> hadoop jar ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input output6
14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/28 13:40:00 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process : 1
14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
14/06/28 13:40:00 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local28288787_0001
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task: attempt_local28288787_0001_m_000000_0
14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/06/28 13:40:01 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
.....
However, the above job does not show up at the web UI :
http://localhost:8088/cluster/apps
My configurations are below,
mapred-site.xml :
<configuration>
<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>
</configuration>
-------
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/matmsh/temp/hadoop2/namenode</value>
<description>The name of the default file system.</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/matmsh/temp/hadoop2/datanode</value>
<description>The name of the default file system.</description>
</property>
</configuration>
----------------
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
------------------------------------
What other configurations are needed to show the job in the web UI ?
Thanks in advance for any assistance !
Shing
Re: Hadoop 2.2.0 : Job not showing on resource management web UI
Posted by Zhijie Shen <zs...@hortonworks.com>.
You were still running the local MR job, which is why you didn't see the
job on RM web UI:
<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>
This config is not in bad format. Please change to "*name*->
mapreduce.framework.name and *value*->yarn".
On Sat, Jun 28, 2014 at 8:54 PM, Shing Hing Man <ma...@yahoo.com> wrote:
> Hi,
> I have set up a single node Hadoop 2.2.0 .
> I can submit wordcount hadoop job to YARN from the command line.
> It run successfully and I checked the result is in hdfs.
> matmsh@gauss:~> hadoop jar
> ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount input output6
> 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
> 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> 14/06/28 13:40:00 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class
> is deprecated. Instead, use mapreduce.job.combine.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class
> is deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local28288787_0001
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task:
> attempt_local28288787_0001_m_000000_0
> 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree :
> [ ]
> 14/06/28 13:40:01 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
> 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> .....
>
> However, the above job does not show up at the web UI :
> http://localhost:8088/cluster/apps
>
> My configurations are below,
> mapred-site.xml :
> <configuration>
>
> <property>
> <value>mapreduce.framework.name</value>
> <name>yarn</name>
> </property>
>
> </configuration>
>
> -------
> hdfs-site.xml
>
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/namenode</value>
> <description>The name of the default file system.</description>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/datanode</value>
> <description>The name of the default file system.</description>
> </property>
>
> </configuration>
> ----------------
> yarn-site.xml
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
>
> </configuration>
>
> ------------------------------------
>
> What other configurations are needed to show the job in the web UI ?
> Thanks in advance for any assistance !
>
> Shing
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop 2.2.0 : Job not showing on resource management web UI
Posted by Zhijie Shen <zs...@hortonworks.com>.
You were still running the local MR job, which is why you didn't see the
job on RM web UI:
<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>
This config is not in bad format. Please change to "*name*->
mapreduce.framework.name and *value*->yarn".
On Sat, Jun 28, 2014 at 8:54 PM, Shing Hing Man <ma...@yahoo.com> wrote:
> Hi,
> I have set up a single node Hadoop 2.2.0 .
> I can submit wordcount hadoop job to YARN from the command line.
> It run successfully and I checked the result is in hdfs.
> matmsh@gauss:~> hadoop jar
> ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount input output6
> 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
> 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> 14/06/28 13:40:00 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class
> is deprecated. Instead, use mapreduce.job.combine.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class
> is deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local28288787_0001
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task:
> attempt_local28288787_0001_m_000000_0
> 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree :
> [ ]
> 14/06/28 13:40:01 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
> 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> .....
>
> However, the above job does not show up at the web UI :
> http://localhost:8088/cluster/apps
>
> My configurations are below,
> mapred-site.xml :
> <configuration>
>
> <property>
> <value>mapreduce.framework.name</value>
> <name>yarn</name>
> </property>
>
> </configuration>
>
> -------
> hdfs-site.xml
>
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/namenode</value>
> <description>The name of the default file system.</description>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/datanode</value>
> <description>The name of the default file system.</description>
> </property>
>
> </configuration>
> ----------------
> yarn-site.xml
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
>
> </configuration>
>
> ------------------------------------
>
> What other configurations are needed to show the job in the web UI ?
> Thanks in advance for any assistance !
>
> Shing
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop 2.2.0 : Job not showing on resource management web UI
Posted by Zhijie Shen <zs...@hortonworks.com>.
You were still running the local MR job, which is why you didn't see the
job on RM web UI:
<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>
This config is not in bad format. Please change to "*name*->
mapreduce.framework.name and *value*->yarn".
On Sat, Jun 28, 2014 at 8:54 PM, Shing Hing Man <ma...@yahoo.com> wrote:
> Hi,
> I have set up a single node Hadoop 2.2.0 .
> I can submit wordcount hadoop job to YARN from the command line.
> It run successfully and I checked the result is in hdfs.
> matmsh@gauss:~> hadoop jar
> ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount input output6
> 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
> 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> 14/06/28 13:40:00 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class
> is deprecated. Instead, use mapreduce.job.combine.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class
> is deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local28288787_0001
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task:
> attempt_local28288787_0001_m_000000_0
> 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree :
> [ ]
> 14/06/28 13:40:01 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
> 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> .....
>
> However, the above job does not show up at the web UI :
> http://localhost:8088/cluster/apps
>
> My configurations are below,
> mapred-site.xml :
> <configuration>
>
> <property>
> <value>mapreduce.framework.name</value>
> <name>yarn</name>
> </property>
>
> </configuration>
>
> -------
> hdfs-site.xml
>
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/namenode</value>
> <description>The name of the default file system.</description>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/datanode</value>
> <description>The name of the default file system.</description>
> </property>
>
> </configuration>
> ----------------
> yarn-site.xml
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
>
> </configuration>
>
> ------------------------------------
>
> What other configurations are needed to show the job in the web UI ?
> Thanks in advance for any assistance !
>
> Shing
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop 2.2.0 : Job not showing on resource management web UI
Posted by Zhijie Shen <zs...@hortonworks.com>.
You were still running the local MR job, which is why you didn't see the
job on RM web UI:
<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>
This config is not in bad format. Please change to "*name*->
mapreduce.framework.name and *value*->yarn".
On Sat, Jun 28, 2014 at 8:54 PM, Shing Hing Man <ma...@yahoo.com> wrote:
> Hi,
> I have set up a single node Hadoop 2.2.0 .
> I can submit wordcount hadoop job to YARN from the command line.
> It run successfully and I checked the result is in hdfs.
> matmsh@gauss:~> hadoop jar
> ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount input output6
> 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
> 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> 14/06/28 13:40:00 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class
> is deprecated. Instead, use mapreduce.job.combine.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class
> is deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local28288787_0001
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 14/06/28 13:40:00 WARN conf.Configuration:
> file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task:
> attempt_local28288787_0001_m_000000_0
> 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree :
> [ ]
> 14/06/28 13:40:01 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
> 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> .....
>
> However, the above job does not show up at the web UI :
> http://localhost:8088/cluster/apps
>
> My configurations are below,
> mapred-site.xml :
> <configuration>
>
> <property>
> <value>mapreduce.framework.name</value>
> <name>yarn</name>
> </property>
>
> </configuration>
>
> -------
> hdfs-site.xml
>
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/namenode</value>
> <description>The name of the default file system.</description>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:/home/matmsh/temp/hadoop2/datanode</value>
> <description>The name of the default file system.</description>
> </property>
>
> </configuration>
> ----------------
> yarn-site.xml
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
>
> </configuration>
>
> ------------------------------------
>
> What other configurations are needed to show the job in the web UI ?
> Thanks in advance for any assistance !
>
> Shing
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.