hadoop without disk i/o

[juber patel <ju...@gmail.com>]

Hi,

Is it possible to use hadoop and not use disk i/o, apart from the
initial input?

I am asking this with the assumption that disk i/o is the bottleneck
in overall processing, even more than the network access if you are on
a dedicated, high speed cluster. (Does anyone have experience to
confirm or reject this assumption?)

I know that my programs logic does not require disk access after
initial input. I don't even require sorting, but would like to combine
the mapper output to reduce its size. This output is fed to another
job/standalone program where it is interpreted meaningfully. I know
this job/standalone program could be the reducer, but I don't want to
spend time in sorting, especially involving disk spills. It is not
required.

Does anyone have a suggestion for this scenario? Is there something
like NetworkInputFormat? Is there a way to start the reduce phase as
mapper output starts coming in? I am thinking in terms of blocking
queues, without disk access but with hadoop's fault tolerance, input
splitting etc.


thanks in advance,

Juber

Re: hadoop without disk i/o

[juber patel <ju...@gmail.com>]

Hi Gaurav,

I do need parallel processing for the part that I am doing in the
mapper right now, and then the results are aggregated. That part of
the problem is embarrassingly parallel and data is so huge it can't be
processed on a single machine.

Juber


On Fri, Jul 30, 2010 at 10:27 AM, Gaurav Sharma
<ga...@gmail.com> wrote:
> Juber - I am wondering if you really do need MR/hadoop for this use case? As
> you have also indicated, a standalone method might be able to do the job for
> you.
>
> On Fri, Jul 30, 2010 at 1:45 AM, juber patel <ju...@gmail.com> wrote:
>>
>> thanks Ken for the reply.
>>
>> well, that's what I am doing right now. But the output from mappers
>> needs to be processed together. Due to the nature of the problem,
>> sorting is trivial once the map output becomes available. That's why I
>> don't want to spend time in hadoop's inbuilt sort that involves disk
>> access. (My sort will be in memory. The number of all possible sort
>> keys is already known).
>>
>> Another problem is that this output needs to be processed further. So,
>> there is unnecessary disk write and read. Again, my premise is that
>> will be the slowest part of the program. So, I want a single reducer
>> or a standalone program that will further process the map output, but
>> I want to avoid disk i/o.
>>
>> thanks,
>>
>> Juber
>>
>>
>> On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <ke...@gmail.com>
>> wrote:
>> > If you don't need sorted input, then you probably don't even need a
>> > reducer. Try putting all your functionality in the mapper and then set
>> > reduce tasks to zero.
>> >
>> > On Thursday, July 29, 2010, juber patel <ju...@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> Is it possible to use hadoop and not use disk i/o, apart from the
>> >> initial input?
>> >>
>> >> I am asking this with the assumption that disk i/o is the bottleneck
>> >> in overall processing, even more than the network access if you are on
>> >> a dedicated, high speed cluster. (Does anyone have experience to
>> >> confirm or reject this assumption?)
>> >>
>> >> I know that my programs logic does not require disk access after
>> >> initial input. I don't even require sorting, but would like to combine
>> >> the mapper output to reduce its size. This output is fed to another
>> >> job/standalone program where it is interpreted meaningfully. I know
>> >> this job/standalone program could be the reducer, but I don't want to
>> >> spend time in sorting, especially involving disk spills. It is not
>> >> required.
>> >>
>> >> Does anyone have a suggestion for this scenario? Is there something
>> >> like NetworkInputFormat? Is there a way to start the reduce phase as
>> >> mapper output starts coming in? I am thinking in terms of blocking
>> >> queues, without disk access but with hadoop's fault tolerance, input
>> >> splitting etc.
>> >>
>> >>
>> >> thanks in advance,
>> >>
>> >> Juber
>> >>
>> >
>
>

Re: hadoop without disk i/o

[Gaurav Sharma <ga...@gmail.com>]

Juber - I am wondering if you really do need MR/hadoop for this use case? As
you have also indicated, a standalone method might be able to do the job for
you.

On Fri, Jul 30, 2010 at 1:45 AM, juber patel <ju...@gmail.com> wrote:

> thanks Ken for the reply.
>
> well, that's what I am doing right now. But the output from mappers
> needs to be processed together. Due to the nature of the problem,
> sorting is trivial once the map output becomes available. That's why I
> don't want to spend time in hadoop's inbuilt sort that involves disk
> access. (My sort will be in memory. The number of all possible sort
> keys is already known).
>
> Another problem is that this output needs to be processed further. So,
> there is unnecessary disk write and read. Again, my premise is that
> will be the slowest part of the program. So, I want a single reducer
> or a standalone program that will further process the map output, but
> I want to avoid disk i/o.
>
> thanks,
>
> Juber
>
>
> On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <ke...@gmail.com>
> wrote:
> > If you don't need sorted input, then you probably don't even need a
> > reducer. Try putting all your functionality in the mapper and then set
> > reduce tasks to zero.
> >
> > On Thursday, July 29, 2010, juber patel <ju...@gmail.com> wrote:
> >> Hi,
> >>
> >> Is it possible to use hadoop and not use disk i/o, apart from the
> >> initial input?
> >>
> >> I am asking this with the assumption that disk i/o is the bottleneck
> >> in overall processing, even more than the network access if you are on
> >> a dedicated, high speed cluster. (Does anyone have experience to
> >> confirm or reject this assumption?)
> >>
> >> I know that my programs logic does not require disk access after
> >> initial input. I don't even require sorting, but would like to combine
> >> the mapper output to reduce its size. This output is fed to another
> >> job/standalone program where it is interpreted meaningfully. I know
> >> this job/standalone program could be the reducer, but I don't want to
> >> spend time in sorting, especially involving disk spills. It is not
> >> required.
> >>
> >> Does anyone have a suggestion for this scenario? Is there something
> >> like NetworkInputFormat? Is there a way to start the reduce phase as
> >> mapper output starts coming in? I am thinking in terms of blocking
> >> queues, without disk access but with hadoop's fault tolerance, input
> >> splitting etc.
> >>
> >>
> >> thanks in advance,
> >>
> >> Juber
> >>
> >
>

Re: hadoop without disk i/o

[juber patel <ju...@gmail.com>]

thanks Ken for the reply.

well, that's what I am doing right now. But the output from mappers
needs to be processed together. Due to the nature of the problem,
sorting is trivial once the map output becomes available. That's why I
don't want to spend time in hadoop's inbuilt sort that involves disk
access. (My sort will be in memory. The number of all possible sort
keys is already known).

Another problem is that this output needs to be processed further. So,
there is unnecessary disk write and read. Again, my premise is that
will be the slowest part of the program. So, I want a single reducer
or a standalone program that will further process the map output, but
I want to avoid disk i/o.

thanks,

Juber


On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <ke...@gmail.com> wrote:
> If you don't need sorted input, then you probably don't even need a
> reducer. Try putting all your functionality in the mapper and then set
> reduce tasks to zero.
>
> On Thursday, July 29, 2010, juber patel <ju...@gmail.com> wrote:
>> Hi,
>>
>> Is it possible to use hadoop and not use disk i/o, apart from the
>> initial input?
>>
>> I am asking this with the assumption that disk i/o is the bottleneck
>> in overall processing, even more than the network access if you are on
>> a dedicated, high speed cluster. (Does anyone have experience to
>> confirm or reject this assumption?)
>>
>> I know that my programs logic does not require disk access after
>> initial input. I don't even require sorting, but would like to combine
>> the mapper output to reduce its size. This output is fed to another
>> job/standalone program where it is interpreted meaningfully. I know
>> this job/standalone program could be the reducer, but I don't want to
>> spend time in sorting, especially involving disk spills. It is not
>> required.
>>
>> Does anyone have a suggestion for this scenario? Is there something
>> like NetworkInputFormat? Is there a way to start the reduce phase as
>> mapper output starts coming in? I am thinking in terms of blocking
>> queues, without disk access but with hadoop's fault tolerance, input
>> splitting etc.
>>
>>
>> thanks in advance,
>>
>> Juber
>>
>

Re: hadoop without disk i/o

[Ken Goodhope <ke...@gmail.com>]

If you don't need sorted input, then you probably don't even need a
reducer. Try putting all your functionality in the mapper and then set
reduce tasks to zero.

On Thursday, July 29, 2010, juber patel <ju...@gmail.com> wrote:
> Hi,
>
> Is it possible to use hadoop and not use disk i/o, apart from the
> initial input?
>
> I am asking this with the assumption that disk i/o is the bottleneck
> in overall processing, even more than the network access if you are on
> a dedicated, high speed cluster. (Does anyone have experience to
> confirm or reject this assumption?)
>
> I know that my programs logic does not require disk access after
> initial input. I don't even require sorting, but would like to combine
> the mapper output to reduce its size. This output is fed to another
> job/standalone program where it is interpreted meaningfully. I know
> this job/standalone program could be the reducer, but I don't want to
> spend time in sorting, especially involving disk spills. It is not
> required.
>
> Does anyone have a suggestion for this scenario? Is there something
> like NetworkInputFormat? Is there a way to start the reduce phase as
> mapper output starts coming in? I am thinking in terms of blocking
> queues, without disk access but with hadoop's fault tolerance, input
> splitting etc.
>
>
> thanks in advance,
>
> Juber
>