where to get the statistics of the job

[Gang Luo <lg...@yahoo.com.cn>]

Hi all,
when the job finish, it shows us the statistics of the job, e.g. map output records, shuffle bytes. How to get this information in my program? Thanks

 --Gang


----- 原始邮件 ----
发件人： pavel kolodin <pa...@gmail.com>
收件人： common-user@hadoop.apache.org
发送日期： 2009/12/1 (周二) 1:34:20 下午
主   题： Re: "dfsadmin -report" says that i need "Superuser privilege". What? :)

On Tue, 01 Dec 2009 12:46:44 -0000, Brian Bockelman <bb...@cse.unl.edu> wrote:

> Hey Pavel,
> 
> The "admin" / superuser for hadoop is the username who launched the daemons.
> 
> So, if you do "ps faux | grep hadoop", you should see some Java processes; the first column of the output tells you what user they are running as.  That user is the super user.
> 
> Since you are already trying this as user "hadoop", I'm guessing that you need to run the command as root.
> 

I am scared to start hadoop as root. Never tried this before (-; Don't understand logic that should make me do that... )
I had experience of launching Hadoop on other 2 couples of machines. I always started all processes as "hadoop".

When i was writing this message, my hadoop distro was compiled by user "root". Now i recompuled it as user "hadoop" and result is the same.

Also i tried start that report request as root - result the same.

hadoopmaster hadoop # hadoop-0.20.1/bin/hadoop dfsadmin -report
report: org.apache.hadoop.security.AccessControlException: Superuser privilege is required
hadoopmaster hadoop #

I had beed using hadoop-0.20.1 on other couple of machines, result was another, this "-report" command was working.

Also, now i am experimenting with 2 VPS hosts, 64 bit, gentoo. Previous machines (i was successful with) was 32 bit, not VPS )

Thank you for answer.



      ___________________________________________________________ 
  好玩贺卡等你发，邮箱贺卡全新上线！ 
http://card.mail.cn.yahoo.com/

Re: where to get the statistics of the job

[Philip Zeyliger <ph...@cloudera.com>]

2009/12/1 Gang Luo <lg...@yahoo.com.cn>

> Hi all,
> when the job finish, it shows us the statistics of the job, e.g. map output
> records, shuffle bytes. How to get this information in my program? Thanks
>

JobClient.getJob(...).getCounters()?

See
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/JobClient.html

-- Philip