displaying intermediate results of map/reduce

[Prasan Ary <vo...@yahoo.com>]

Hi All,
  I am using eclipse to write a map/reduce java application that connects to hadoop on remote cluster.
  Is there a way I can display intermediate results of map ( or reduce) much the same way as I would use System.out.println( variable_name) if I were running any application on a single machine?
   
  thx,
  Prasan.

       
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it now.

Re: displaying intermediate results of map/reduce

[Owen O'Malley <oo...@yahoo-inc.com>]

On Mar 6, 2008, at 8:26 AM, Enis Soztutar wrote:

> You can also run the job in local mode with zero reducers, so that  
> the map results are the results of the job.

This is a good suggestion. Running with TextOutputFormat and reduces  
= 0 will work great on the cluster too. You'll be able to see what  
your map functions are producing very easily. *smile*

-- wen

Re: displaying intermediate results of map/reduce

[Enis Soztutar <en...@gmail.com>]

You can also run the job in local mode with zero reducers, so that the 
map results are the results of the job.

Prasan Ary wrote:
> Hi All,
>   I am using eclipse to write a map/reduce java application that connects to hadoop on remote cluster.
>   Is there a way I can display intermediate results of map ( or reduce) much the same way as I would use System.out.println( variable_name) if I were running any application on a single machine?
>    
>   thx,
>   Prasan.
>
>        
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it now.
>   

Re: displaying intermediate results of map/reduce

[Ted Dunning <td...@veoh.com>]

You can use System.out if you like and then look at the results as each map
or reduce completes via the web administration tool.

Also, you can use counters via the reporter passed to your map and reduce
classes to get immediate feedback.


On 3/6/08 8:12 AM, "Prasan Ary" <vo...@yahoo.com> wrote:

> Hi All,
>   I am using eclipse to write a map/reduce java application that connects to
> hadoop on remote cluster.
>   Is there a way I can display intermediate results of map ( or reduce) much
> the same way as I would use System.out.println( variable_name) if I were
> running any application on a single machine?
>    
>   thx,
>   Prasan.
> 
>        
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it
> now.