You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by Ahmad Humayun <ah...@gmail.com> on 2008/03/12 13:25:31 UTC

MapTask intermediate data

Can somebody explain the process of the MapTask writing to the file the
intermediate output. The MapOutputBuffer.mergeParts() seems to be writing
different buffers into the same file. It seems like its also keeping an
index file.

It will be great if somebody can guide me through the process.


regards,
-- 
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315

Re: MapTask intermediate data

Posted by Ahmad Humayun <ah...@gmail.com>.

Thanks Amar :)

On Wed, Mar 12, 2008 at 5:48 PM, Amar Kamat <am...@yahoo-inc.com> wrote:

> See HADOOP-2919. It explains the current technique. This will be a good
> starting point.
> Amar
> On Wed, 12 Mar 2008, Ahmad Humayun wrote:
>
> > Can somebody explain the process of the MapTask writing to the file the
> > intermediate output. The MapOutputBuffer.mergeParts() seems to be
> writing
> > different buffers into the same file. It seems like its also keeping an
> > index file.
> >
> > It will be great if somebody can guide me through the process.
> >
> >
> > regards,
> >
>



-- 
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315

Re: MapTask intermediate data

Posted by Amar Kamat <am...@yahoo-inc.com>.

See HADOOP-2919. It explains the current technique. This will be a good 
starting point.
Amar
On Wed, 12 Mar 2008, Ahmad Humayun wrote:

> Can somebody explain the process of the MapTask writing to the file the
> intermediate output. The MapOutputBuffer.mergeParts() seems to be writing
> different buffers into the same file. It seems like its also keeping an
> index file.
>
> It will be great if somebody can guide me through the process.
>
>
> regards,
>