You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Sandy <sn...@gmail.com> on 2009/02/27 20:12:39 UTC

Profiling Hadoop

Hello,

Could anyone recommend any software for profiling the performance of
MapReduce applications one may write for Hadoop? I am currently developing
in Java.

Thanks,
-SM

Re: Profiling Hadoop

Posted by Sandy <sn...@gmail.com>.

Thanks, Bryan! I'll check out YourKit :-)
-SM

On Fri, Feb 27, 2009 at 1:55 PM, Bryan Duxbury <br...@rapleaf.com> wrote:

> I've used YourKit Java Profiler pretty successfully. There's a JobConf
> parameter you can flip on that will cause a few maps and reduces to start
> with profiling on, so you won't be overwhelmed with info.
>
> -Bryan
>
>
> On Feb 27, 2009, at 11:12 AM, Sandy wrote:
>
>  Hello,
>>
>> Could anyone recommend any software for profiling the performance of
>> MapReduce applications one may write for Hadoop? I am currently developing
>> in Java.
>>
>> Thanks,
>> -SM
>>
>
>

discp problem: global counters are inaccurate

Posted by Steve Kuo <St...@eharmony.com>.

The following distcp command has been working fine until today.  Does anyone have any idea on what does "global counters are inaccurate" mean?

~/projects/pig] hadoop distcp s3n://$ACCESS_KEY_ID:$SECRET_ACCESS_KEY_ID@$BUCKET_AMF/ hdfs://hadoop01.dc1.foo.com:9000/user/bar/data/amfdata
With failures, global counters are inaccurate; consider running with -i Copy failed: java.lang.NullPointerException
        at org.apache.hadoop.tools.DistCp.makeRelative(DistCp.java:801)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:979)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:614)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:764)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:784)

Re: Profiling Hadoop

Posted by Bryan Duxbury <br...@rapleaf.com>.

I've used YourKit Java Profiler pretty successfully. There's a  
JobConf parameter you can flip on that will cause a few maps and  
reduces to start with profiling on, so you won't be overwhelmed with  
info.

-Bryan

On Feb 27, 2009, at 11:12 AM, Sandy wrote:

> Hello,
>
> Could anyone recommend any software for profiling the performance of
> MapReduce applications one may write for Hadoop? I am currently  
> developing
> in Java.
>
> Thanks,
> -SM