You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Alexandros Konstantinakis - Karmis <ae...@ece.ntua.gr> on 2010/06/04 13:30:21 UTC

Profiling hadoop application/MultipleOutputs

Hello everyone.

i just have a couple of short questions. I am using the new API.

a) i need to profile my application. I am aware of the setProfileEnabled 
method in JobConf, has this been ported to the new API yet?

I have used PerfAnal in the past with great success for non-hadoop java 
applications. Is it possible to use that to read the profile data?


b) I need to output 1 file per reducer. I understand this is possible 
with multipleOutputs or MultiOutputFormat, neither of which has been 
ported to the new API. Is my best choice to use this patch 
https://issues.apache.org/jira/browse/MAPREDUCE-370 ?

thanks in advance

Re: Profiling hadoop application/MultipleOutputs

Posted by Alex Kozlov <al...@cloudera.com>.

On Fri, Jun 4, 2010 at 4:30 AM, Alexandros Konstantinakis - Karmis <
aeolist@ece.ntua.gr> wrote:

>
>
> b) I need to output 1 file per reducer. I understand this is possible with
> multipleOutputs or MultiOutputFormat, neither of which has been ported to
> the new API. Is my best choice to use this patch
> https://issues.apache.org/jira/browse/MAPREDUCE-370 ?
>
>
Hi Alexander,

A reducer outputs one file per reducer by default.  MultipleOuputs has been
backported to CDH2 <http://archive.cloudera.com/cdh/2/>.

Alex K