HDFS - MapReduce coupling

[Matthew John <tm...@gmail.com>]

Hi all,

1) I wanted to know how strong the coupling between HDFS and MapReduce
(programming abstraction) in Hadoop is. Can someone throw some light on the
protocols used between HDFS and JobTracker/TaskTracker/Namenode
interactions. Any pointer on this would be of great help!

2) Does the Hadoop system utilize the local storage directly for any purpose
(without going through the HDFS) in clustered mode?

Thanks,
Matthew John

Re: HDFS - MapReduce coupling

[James Seigel <ja...@tynt.com>]

If you are pressed for time, you could look at the source code.  I
believe a huge proportion of the people that could answer your
question ( and it isn't a small one ) are sleeping right now. :)

Source code is probably your best answer.

James

Sent from my mobile. Please excuse the typos.

On 2011-05-02, at 5:23 AM, Matthew John <tm...@gmail.com> wrote:

> someone kindly give some pointers on this!!
>
> On Mon, May 2, 2011 at 12:46 PM, Matthew John <tm...@gmail.com>wrote:
>
>> Any documentations on how the different daemons do the write/read on HDFS
>> and Local File System (direct), I mean the different protocols used in the
>> interactions. I basically wanted to figure out how intricate the coupling
>> between the Storage (HDFS + Local) and other processes in the Hadoop
>> infrastructure is.
>>
>>
>>
>> On Mon, May 2, 2011 at 12:26 PM, Ted Dunning <td...@maprtech.com>wrote:
>>
>>> Yes.  There is quite a bit of need for the local file system in clustered
>>> mode.
>>>
>>> For one think, all of the shuffle intermediate files are on local disk.
>>> For
>>> another, the distributed cache is actually stored on local disk.
>>>
>>> HFDS is a frail vessel that cannot cope with all the needs.
>>>
>>> On Sun, May 1, 2011 at 11:48 PM, Matthew John <tmatthewjohn1988@gmail.com
>>>> wrote:
>>>
>>>> ...
>>>> 2) Does the Hadoop system utilize the local storage directly for any
>>>> purpose
>>>> (without going through the HDFS) in clustered mode?
>>>>
>>>>
>>>
>>
>>

Re: HDFS - MapReduce coupling

[Matthew John <tm...@gmail.com>]

someone kindly give some pointers on this!!

On Mon, May 2, 2011 at 12:46 PM, Matthew John <tm...@gmail.com>wrote:

> Any documentations on how the different daemons do the write/read on HDFS
> and Local File System (direct), I mean the different protocols used in the
> interactions. I basically wanted to figure out how intricate the coupling
> between the Storage (HDFS + Local) and other processes in the Hadoop
> infrastructure is.
>
>
>
> On Mon, May 2, 2011 at 12:26 PM, Ted Dunning <td...@maprtech.com>wrote:
>
>> Yes.  There is quite a bit of need for the local file system in clustered
>> mode.
>>
>> For one think, all of the shuffle intermediate files are on local disk.
>>  For
>> another, the distributed cache is actually stored on local disk.
>>
>> HFDS is a frail vessel that cannot cope with all the needs.
>>
>> On Sun, May 1, 2011 at 11:48 PM, Matthew John <tmatthewjohn1988@gmail.com
>> >wrote:
>>
>> > ...
>> > 2) Does the Hadoop system utilize the local storage directly for any
>> > purpose
>> > (without going through the HDFS) in clustered mode?
>> >
>> >
>>
>
>

Re: HDFS - MapReduce coupling

[Matthew John <tm...@gmail.com>]

Any documentations on how the different daemons do the write/read on HDFS
and Local File System (direct), I mean the different protocols used in the
interactions. I basically wanted to figure out how intricate the coupling
between the Storage (HDFS + Local) and other processes in the Hadoop
infrastructure is.


On Mon, May 2, 2011 at 12:26 PM, Ted Dunning <td...@maprtech.com> wrote:

> Yes.  There is quite a bit of need for the local file system in clustered
> mode.
>
> For one think, all of the shuffle intermediate files are on local disk.
>  For
> another, the distributed cache is actually stored on local disk.
>
> HFDS is a frail vessel that cannot cope with all the needs.
>
> On Sun, May 1, 2011 at 11:48 PM, Matthew John <tmatthewjohn1988@gmail.com
> >wrote:
>
> > ...
> > 2) Does the Hadoop system utilize the local storage directly for any
> > purpose
> > (without going through the HDFS) in clustered mode?
> >
> >
>

Re: HDFS - MapReduce coupling

[Ted Dunning <td...@maprtech.com>]

Yes.  There is quite a bit of need for the local file system in clustered
mode.

For one think, all of the shuffle intermediate files are on local disk.  For
another, the distributed cache is actually stored on local disk.

HFDS is a frail vessel that cannot cope with all the needs.

On Sun, May 1, 2011 at 11:48 PM, Matthew John <tm...@gmail.com>wrote:

> ...
> 2) Does the Hadoop system utilize the local storage directly for any
> purpose
> (without going through the HDFS) in clustered mode?
>
>