You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ou senshaw <se...@gmail.com> on 2014/08/14 08:14:39 UTC
Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get them
from client output. I was wondering if there is other place in name node or
data node to get the final counter measures regarding job id?
Thanks,
Shaw
Re: Hadoop 2.2 Built-in Counters
Posted by ou senshaw <se...@gmail.com>.
Thanks guys! It's working.
On Thu, Aug 14, 2014 at 2:39 AM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> This is the content of my shell script for running the job history server:
>
> cd $HADOOP_PREFIX
> hadoop fs -mkdir -p /mr-history/tmp
> hadoop fs -chmod -R 1777 /mr-history/tmp
> hadoop fs -mkdir -p /mr-history/done
> hadoop fs -chmod -R 1777 /mr-history/done
>
> sbin/mr-jobhistory-daemon.sh start historyserver
>
> These configurable variables are in mapred-site.xml
>
> <property>
> <name>mapreduce.jobhistory.address</name>
> <value>137.195.143.129:10020</value>
> <description>Default port is 10020.</description>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>137.195.143.129:19888</value>
> <description>Default port is 19888.</description>
> </property>
>
> I start the history server on the same node as my resource manager
>
>
> The counters are available from when the job is running from:
>
> http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
> job_1408007466921_0002
>
> Drill down through the application master to the job.
>
> If you don¹t have the history server running the job data is not
> persistent.
>
> Hope this helps.
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From: ou senshaw <se...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Thursday, 14 August 2014 07:14
> To: <us...@hadoop.apache.org>
> Subject: Hadoop 2.2 Built-in Counters
>
>
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get
> them from client output. I was wondering if there is other place in name
> node or data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
>
>
>
Re: Hadoop 2.2 Built-in Counters
Posted by ou senshaw <se...@gmail.com>.
Thanks guys! It's working.
On Thu, Aug 14, 2014 at 2:39 AM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> This is the content of my shell script for running the job history server:
>
> cd $HADOOP_PREFIX
> hadoop fs -mkdir -p /mr-history/tmp
> hadoop fs -chmod -R 1777 /mr-history/tmp
> hadoop fs -mkdir -p /mr-history/done
> hadoop fs -chmod -R 1777 /mr-history/done
>
> sbin/mr-jobhistory-daemon.sh start historyserver
>
> These configurable variables are in mapred-site.xml
>
> <property>
> <name>mapreduce.jobhistory.address</name>
> <value>137.195.143.129:10020</value>
> <description>Default port is 10020.</description>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>137.195.143.129:19888</value>
> <description>Default port is 19888.</description>
> </property>
>
> I start the history server on the same node as my resource manager
>
>
> The counters are available from when the job is running from:
>
> http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
> job_1408007466921_0002
>
> Drill down through the application master to the job.
>
> If you don¹t have the history server running the job data is not
> persistent.
>
> Hope this helps.
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From: ou senshaw <se...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Thursday, 14 August 2014 07:14
> To: <us...@hadoop.apache.org>
> Subject: Hadoop 2.2 Built-in Counters
>
>
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get
> them from client output. I was wondering if there is other place in name
> node or data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
>
>
>
Re: Hadoop 2.2 Built-in Counters
Posted by ou senshaw <se...@gmail.com>.
Thanks guys! It's working.
On Thu, Aug 14, 2014 at 2:39 AM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> This is the content of my shell script for running the job history server:
>
> cd $HADOOP_PREFIX
> hadoop fs -mkdir -p /mr-history/tmp
> hadoop fs -chmod -R 1777 /mr-history/tmp
> hadoop fs -mkdir -p /mr-history/done
> hadoop fs -chmod -R 1777 /mr-history/done
>
> sbin/mr-jobhistory-daemon.sh start historyserver
>
> These configurable variables are in mapred-site.xml
>
> <property>
> <name>mapreduce.jobhistory.address</name>
> <value>137.195.143.129:10020</value>
> <description>Default port is 10020.</description>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>137.195.143.129:19888</value>
> <description>Default port is 19888.</description>
> </property>
>
> I start the history server on the same node as my resource manager
>
>
> The counters are available from when the job is running from:
>
> http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
> job_1408007466921_0002
>
> Drill down through the application master to the job.
>
> If you don¹t have the history server running the job data is not
> persistent.
>
> Hope this helps.
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From: ou senshaw <se...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Thursday, 14 August 2014 07:14
> To: <us...@hadoop.apache.org>
> Subject: Hadoop 2.2 Built-in Counters
>
>
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get
> them from client output. I was wondering if there is other place in name
> node or data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
>
>
>
Re: Hadoop 2.2 Built-in Counters
Posted by ou senshaw <se...@gmail.com>.
Thanks guys! It's working.
On Thu, Aug 14, 2014 at 2:39 AM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> This is the content of my shell script for running the job history server:
>
> cd $HADOOP_PREFIX
> hadoop fs -mkdir -p /mr-history/tmp
> hadoop fs -chmod -R 1777 /mr-history/tmp
> hadoop fs -mkdir -p /mr-history/done
> hadoop fs -chmod -R 1777 /mr-history/done
>
> sbin/mr-jobhistory-daemon.sh start historyserver
>
> These configurable variables are in mapred-site.xml
>
> <property>
> <name>mapreduce.jobhistory.address</name>
> <value>137.195.143.129:10020</value>
> <description>Default port is 10020.</description>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>137.195.143.129:19888</value>
> <description>Default port is 19888.</description>
> </property>
>
> I start the history server on the same node as my resource manager
>
>
> The counters are available from when the job is running from:
>
> http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
> job_1408007466921_0002
>
> Drill down through the application master to the job.
>
> If you don¹t have the history server running the job data is not
> persistent.
>
> Hope this helps.
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From: ou senshaw <se...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Thursday, 14 August 2014 07:14
> To: <us...@hadoop.apache.org>
> Subject: Hadoop 2.2 Built-in Counters
>
>
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get
> them from client output. I was wondering if there is other place in name
> node or data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
>
>
>
Re: Hadoop 2.2 Built-in Counters
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
This is the content of my shell script for running the job history server:
cd $HADOOP_PREFIX
hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done
sbin/mr-jobhistory-daemon.sh start historyserver
These configurable variables are in mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>137.195.143.129:10020</value>
<description>Default port is 10020.</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>137.195.143.129:19888</value>
<description>Default port is 19888.</description>
</property>
I start the history server on the same node as my resource manager
The counters are available from when the job is running from:
http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
job_1408007466921_0002
Drill down through the application master to the job.
If you don¹t have the history server running the job data is not
persistent.
Hope this helps.
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: ou senshaw <se...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Thursday, 14 August 2014 07:14
To: <us...@hadoop.apache.org>
Subject: Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?
Thanks,
Shaw
Re: Hadoop 2.2 Built-in Counters
Posted by Harsh J <ha...@cloudera.com>.
You can see them in the MR Job HistoryServer UI if you have one
configured. You can find config properties and launch command in
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/ClusterSetup.html
On Thu, Aug 14, 2014 at 11:44 AM, ou senshaw <se...@gmail.com> wrote:
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get them
> from client output. I was wondering if there is other place in name node or
> data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
--
Harsh J
Re: Hadoop 2.2 Built-in Counters
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
This is the content of my shell script for running the job history server:
cd $HADOOP_PREFIX
hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done
sbin/mr-jobhistory-daemon.sh start historyserver
These configurable variables are in mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>137.195.143.129:10020</value>
<description>Default port is 10020.</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>137.195.143.129:19888</value>
<description>Default port is 19888.</description>
</property>
I start the history server on the same node as my resource manager
The counters are available from when the job is running from:
http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
job_1408007466921_0002
Drill down through the application master to the job.
If you don¹t have the history server running the job data is not
persistent.
Hope this helps.
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: ou senshaw <se...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Thursday, 14 August 2014 07:14
To: <us...@hadoop.apache.org>
Subject: Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?
Thanks,
Shaw
Re: Hadoop 2.2 Built-in Counters
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
This is the content of my shell script for running the job history server:
cd $HADOOP_PREFIX
hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done
sbin/mr-jobhistory-daemon.sh start historyserver
These configurable variables are in mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>137.195.143.129:10020</value>
<description>Default port is 10020.</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>137.195.143.129:19888</value>
<description>Default port is 19888.</description>
</property>
I start the history server on the same node as my resource manager
The counters are available from when the job is running from:
http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
job_1408007466921_0002
Drill down through the application master to the job.
If you don¹t have the history server running the job data is not
persistent.
Hope this helps.
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: ou senshaw <se...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Thursday, 14 August 2014 07:14
To: <us...@hadoop.apache.org>
Subject: Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?
Thanks,
Shaw
Re: Hadoop 2.2 Built-in Counters
Posted by Harsh J <ha...@cloudera.com>.
You can see them in the MR Job HistoryServer UI if you have one
configured. You can find config properties and launch command in
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/ClusterSetup.html
On Thu, Aug 14, 2014 at 11:44 AM, ou senshaw <se...@gmail.com> wrote:
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get them
> from client output. I was wondering if there is other place in name node or
> data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
--
Harsh J
Re: Hadoop 2.2 Built-in Counters
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
This is the content of my shell script for running the job history server:
cd $HADOOP_PREFIX
hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done
sbin/mr-jobhistory-daemon.sh start historyserver
These configurable variables are in mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>137.195.143.129:10020</value>
<description>Default port is 10020.</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>137.195.143.129:19888</value>
<description>Default port is 19888.</description>
</property>
I start the history server on the same node as my resource manager
The counters are available from when the job is running from:
http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
job_1408007466921_0002
Drill down through the application master to the job.
If you don¹t have the history server running the job data is not
persistent.
Hope this helps.
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: ou senshaw <se...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Thursday, 14 August 2014 07:14
To: <us...@hadoop.apache.org>
Subject: Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?
Thanks,
Shaw
Re: Hadoop 2.2 Built-in Counters
Posted by Harsh J <ha...@cloudera.com>.
You can see them in the MR Job HistoryServer UI if you have one
configured. You can find config properties and launch command in
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/ClusterSetup.html
On Thu, Aug 14, 2014 at 11:44 AM, ou senshaw <se...@gmail.com> wrote:
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get them
> from client output. I was wondering if there is other place in name node or
> data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
--
Harsh J
Re: Hadoop 2.2 Built-in Counters
Posted by Harsh J <ha...@cloudera.com>.
You can see them in the MR Job HistoryServer UI if you have one
configured. You can find config properties and launch command in
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/ClusterSetup.html
On Thu, Aug 14, 2014 at 11:44 AM, ou senshaw <se...@gmail.com> wrote:
> Hi all,
> I'm trying to analyze my mapreduce job performance via built-in counters
> such as physical memory usage, heap memory usage...
> When the job is running, I can watch these counters via Resource manager
> website(namenode:8088). However, when the job is done, counter information
> is not available in resource manager website anymore. I know I can get them
> from client output. I was wondering if there is other place in name node or
> data node to get the final counter measures regarding job id?
> Thanks,
> Shaw
--
Harsh J