access to part file in reducer

[abc xyz <fa...@yahoo.com>]

Hi,

Is it possible to get access to the part file generated by a reducer in the 
close function of the reducer for some manipulations and then writing it back? 
If yes, how can I determine the name of the part-file and how can I access it?

Thanks


      

Re: access to part file in reducer

[Gang Luo <lg...@yahoo.com.cn>]

I think when close() is called, part of the data is in the output buffer, the 
file is not complete and it doesn't make sense to access that file at this time, 
unless you can wait until everything is done. 


If you want to manipulate the data, why not defer outputting? You don't output 
anything in map() but just buffer the result, do whatever you want in close() 
then output all of them afterward. Of course, you have to ensure your have 
enough memory.

-Gang




----- 原始邮件 ----
发件人： abc xyz <fa...@yahoo.com>
收件人： common-user@hadoop.apache.org
发送日期： 2010/7/17 (周六) 12:19:47 下午
主   题： access to part file in reducer

Hi,

Is it possible to get access to the part file generated by a reducer in the 
close function of the reducer for some manipulations and then writing it back? 
If yes, how can I determine the name of the part-file and how can I access it?

Thanks