You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by tom <zs...@gmail.com> on 2009/11/29 04:55:51 UTC
what is the major difference between Hadoop and cloudMapReduce?
Hi,
I know Hadoop is a suite of software service for the cluster
computing, including mapreduce implementation, distributed database,
distributed file system, etc.
the confusion to me is If cloudMapReduce just is focused on Cloud
Computing Infrastructure and must be depend on overall distributed
service provided and hidden by Cloud Computing Infrastructure?
And, cloudMapReduce just can be a alternate implementation of MapReduce?
Is it right?
Best regards
Tom
RE: what is the major difference between Hadoop and cloudMapReduce?
Posted by hu...@accenture.com.
Todd,
Yes, the difference is that, if special ordering is required, we hold a few keys' worth of values in memory v.s. Hadoop holding one key's worth of values.
SQS has no limit on the number of queues you can create and the number of messages you can put in a queue. In practice, we have run an experiment in which we created 4,000 queues with 100 threads, and it finished successfully in a few seconds. Have not tried to stress test to millions of queues though. But even if the test fails with millions of queues, it will be a bug with SQS to be fixed since Amazon advertises that there is no limit. Thanks.
-Huan
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, December 03, 2009 9:45 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>
> On Thu, Dec 3, 2009 at 9:39 AM, <hu...@accenture.com> wrote:
>
> > > -----Original Message-----
> > > From: Todd Lipcon [mailto:todd@cloudera.com]
> > > Sent: Monday, November 30, 2009 8:15 AM
> > > To: general@hadoop.apache.org
> > > Subject: Re: what is the major difference between Hadoop and
> > > cloudMapReduce?
> > >
> > > On Mon, Nov 30, 2009 at 1:48 AM, <hu...@accenture.com> wrote:
> > >
> > > > Todd,
> > > >
> > > > We do not keep all values for a key in memory. Instead, we only
> keep
> > > the
> > > > partial reduce result in memory, but throw away the value as soon
> as
> > > it is
> > > > used. The point you raised is still very valid if the reduce
> state
> > > > maintained per key is large, which I hope is a very rare case. If
> you
> > > have
> > > > some concrete workload examples, it will help us prioritize the
> > > development
> > > > effort. I can definitely see the benefits of introducing a paging
> > > mechanism
> > > > to spill partial reduce results to the output SQS queue in the
> future.
> > > > Thanks.
> > > >
> > >
> > > Hi Huan,
> > >
> > > I guess I misremembered or misread the paper.
> > >
> > > Given this technique, doesn't it mean that reducers can only work
> when
> > > commutative and associative?
> > >
> > > -Todd
> >
> > Todd,
> >
> > I do not see how it is different from Hadoop's iterator interface,
> unless
> > the reduce function relies on the fact that the values are sorted in
> a
> > particular order when fed by the iterator one at a time.
> >
> > If there is no assumption on the value ordering, or the ordering
> expected
> > is different from what the iterator presents, the reduce function has
> to
> > read in all values from the iterator first (page to disk if
> necessary),
> > rearrange them as necessary, then process based on that new ordering.
> This
> > will be the same as what we will do in our iterator interface. In the
> next()
> > function, our reduce function can read in all values from the
> iterator (page
> > to disk if necessary), then in the finish() function, our reduce
> function
> > rearranges the ordering and process based on the new ordering.
> >
> >
> If you want sorted values, you can get that in Hadoop, though it's not
> on by
> default.
>
> Also, the reducer only needs to keep all the values for a single key in
> RAM
> in this case. In your case, since the keys come in any order, the
> reducer
> would have to keep all the values for every key in that partition in
> RAM. I
> guess you're suggesting that you could have enough partitions that
> "every
> key in that partition" is only one or two, but for large datasets this
> just
> doesn't soudn feasible (can you get millions of SQS queues?)
>
> Sure, you can implement your own paging to disk, and then an external
> sort,
> and then read them back in the more convenient sorted order. But then
> you're
> just implementing what Hadoop already does for you :)
>
> -Todd
>
>
> >
> > This message is for the designated recipient only and may contain
> > privileged, proprietary, or otherwise private information. If you
> have
> > received it in error, please notify the sender immediately and delete
> the
> > original. Any other use of the email by you is prohibited.
> >
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Dec 3, 2009 at 9:39 AM, <hu...@accenture.com> wrote:
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Monday, November 30, 2009 8:15 AM
> > To: general@hadoop.apache.org
> > Subject: Re: what is the major difference between Hadoop and
> > cloudMapReduce?
> >
> > On Mon, Nov 30, 2009 at 1:48 AM, <hu...@accenture.com> wrote:
> >
> > > Todd,
> > >
> > > We do not keep all values for a key in memory. Instead, we only keep
> > the
> > > partial reduce result in memory, but throw away the value as soon as
> > it is
> > > used. The point you raised is still very valid if the reduce state
> > > maintained per key is large, which I hope is a very rare case. If you
> > have
> > > some concrete workload examples, it will help us prioritize the
> > development
> > > effort. I can definitely see the benefits of introducing a paging
> > mechanism
> > > to spill partial reduce results to the output SQS queue in the future.
> > > Thanks.
> > >
> >
> > Hi Huan,
> >
> > I guess I misremembered or misread the paper.
> >
> > Given this technique, doesn't it mean that reducers can only work when
> > commutative and associative?
> >
> > -Todd
>
> Todd,
>
> I do not see how it is different from Hadoop's iterator interface, unless
> the reduce function relies on the fact that the values are sorted in a
> particular order when fed by the iterator one at a time.
>
> If there is no assumption on the value ordering, or the ordering expected
> is different from what the iterator presents, the reduce function has to
> read in all values from the iterator first (page to disk if necessary),
> rearrange them as necessary, then process based on that new ordering. This
> will be the same as what we will do in our iterator interface. In the next()
> function, our reduce function can read in all values from the iterator (page
> to disk if necessary), then in the finish() function, our reduce function
> rearranges the ordering and process based on the new ordering.
>
>
If you want sorted values, you can get that in Hadoop, though it's not on by
default.
Also, the reducer only needs to keep all the values for a single key in RAM
in this case. In your case, since the keys come in any order, the reducer
would have to keep all the values for every key in that partition in RAM. I
guess you're suggesting that you could have enough partitions that "every
key in that partition" is only one or two, but for large datasets this just
doesn't soudn feasible (can you get millions of SQS queues?)
Sure, you can implement your own paging to disk, and then an external sort,
and then read them back in the more convenient sorted order. But then you're
just implementing what Hadoop already does for you :)
-Todd
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the email by you is prohibited.
>
RE: what is the major difference between Hadoop and cloudMapReduce?
Posted by hu...@accenture.com.
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Monday, November 30, 2009 8:15 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>
> On Mon, Nov 30, 2009 at 1:48 AM, <hu...@accenture.com> wrote:
>
> > Todd,
> >
> > We do not keep all values for a key in memory. Instead, we only keep
> the
> > partial reduce result in memory, but throw away the value as soon as
> it is
> > used. The point you raised is still very valid if the reduce state
> > maintained per key is large, which I hope is a very rare case. If you
> have
> > some concrete workload examples, it will help us prioritize the
> development
> > effort. I can definitely see the benefits of introducing a paging
> mechanism
> > to spill partial reduce results to the output SQS queue in the future.
> > Thanks.
> >
>
> Hi Huan,
>
> I guess I misremembered or misread the paper.
>
> Given this technique, doesn't it mean that reducers can only work when
> commutative and associative?
>
> -Todd
Todd,
I do not see how it is different from Hadoop's iterator interface, unless the reduce function relies on the fact that the values are sorted in a particular order when fed by the iterator one at a time.
If there is no assumption on the value ordering, or the ordering expected is different from what the iterator presents, the reduce function has to read in all values from the iterator first (page to disk if necessary), rearrange them as necessary, then process based on that new ordering. This will be the same as what we will do in our iterator interface. In the next() function, our reduce function can read in all values from the iterator (page to disk if necessary), then in the finish() function, our reduce function rearranges the ordering and process based on the new ordering.
Apologies for the late reply. Traveling in china with no reliable network connection this week. Thanks.
-Huan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Nov 30, 2009 at 1:48 AM, <hu...@accenture.com> wrote:
> Todd,
>
> We do not keep all values for a key in memory. Instead, we only keep the
> partial reduce result in memory, but throw away the value as soon as it is
> used. The point you raised is still very valid if the reduce state
> maintained per key is large, which I hope is a very rare case. If you have
> some concrete workload examples, it will help us prioritize the development
> effort. I can definitely see the benefits of introducing a paging mechanism
> to spill partial reduce results to the output SQS queue in the future.
> Thanks.
>
Hi Huan,
I guess I misremembered or misread the paper.
Given this technique, doesn't it mean that reducers can only work when
commutative and associative?
-Todd
>
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Sunday, November 29, 2009 10:15 AM
> > To: general@hadoop.apache.org
> > Subject: Re: what is the major difference between Hadoop and
> > cloudMapReduce?
> >
> > Another difference is that Cloud Mapreduce doesn't scale well when
> > there are
> > a large number of values for the same key. They collect in a single SQS
> > queue per partition, and in the reducer reads that into a hashtable
> > that
> > never spills to disk. So, if you have more values for a key than fit in
> > RAM,
> > you can't run a job. Hadoop on the other hand spills reduce input to
> > disk
> > when it's too large to sort in memory. The paper says to add more
> > partitions
> > if you have a lot of data, but it doesn't help in the case of skewed
> > reduce
> > input (very common in a lot of workloads)
> >
> > -Todd
> >
> > On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder
> > <br...@gmail.com>wrote:
> >
> > > On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> > > <ti...@gmail.com> wrote:
> > > > Hi Tom,
> > > >
> > > > One major difference is that Hadoop allows working with any object
> > > > that you can serialize, rather than only Strings so it is
> > applicable
> > > > for use with (e.g) image processing, or rendering to images or PDFs
> > on
> > > > the output format.
> > >
> > > Yes, you noted this on the general@hadoop list. I think your
> > > suggestion to expand this to support other types is a good one.
> > >
> > > Bruce
> > > --
> > > perl -e 'print
> > > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > > );'
> > >
> > > ActiveMQ in Action: http://bit.ly/2je6cQ
> > > Blog: http://bruceblog.org/
> > > Twitter: http://twitter.com/brucesnyder
> > >
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the email by you is prohibited.
>
RE: what is the major difference between Hadoop and cloudMapReduce?
Posted by hu...@accenture.com.
Todd,
We do not keep all values for a key in memory. Instead, we only keep the partial reduce result in memory, but throw away the value as soon as it is used. The point you raised is still very valid if the reduce state maintained per key is large, which I hope is a very rare case. If you have some concrete workload examples, it will help us prioritize the development effort. I can definitely see the benefits of introducing a paging mechanism to spill partial reduce results to the output SQS queue in the future. Thanks.
-Huan
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Sunday, November 29, 2009 10:15 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>
> Another difference is that Cloud Mapreduce doesn't scale well when
> there are
> a large number of values for the same key. They collect in a single SQS
> queue per partition, and in the reducer reads that into a hashtable
> that
> never spills to disk. So, if you have more values for a key than fit in
> RAM,
> you can't run a job. Hadoop on the other hand spills reduce input to
> disk
> when it's too large to sort in memory. The paper says to add more
> partitions
> if you have a lot of data, but it doesn't help in the case of skewed
> reduce
> input (very common in a lot of workloads)
>
> -Todd
>
> On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder
> <br...@gmail.com>wrote:
>
> > On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> > <ti...@gmail.com> wrote:
> > > Hi Tom,
> > >
> > > One major difference is that Hadoop allows working with any object
> > > that you can serialize, rather than only Strings so it is
> applicable
> > > for use with (e.g) image processing, or rendering to images or PDFs
> on
> > > the output format.
> >
> > Yes, you noted this on the general@hadoop list. I think your
> > suggestion to expand this to support other types is a good one.
> >
> > Bruce
> > --
> > perl -e 'print
> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > );'
> >
> > ActiveMQ in Action: http://bit.ly/2je6cQ
> > Blog: http://bruceblog.org/
> > Twitter: http://twitter.com/brucesnyder
> >
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Todd Lipcon <to...@cloudera.com>.
Another difference is that Cloud Mapreduce doesn't scale well when there are
a large number of values for the same key. They collect in a single SQS
queue per partition, and in the reducer reads that into a hashtable that
never spills to disk. So, if you have more values for a key than fit in RAM,
you can't run a job. Hadoop on the other hand spills reduce input to disk
when it's too large to sort in memory. The paper says to add more partitions
if you have a lot of data, but it doesn't help in the case of skewed reduce
input (very common in a lot of workloads)
-Todd
On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder <br...@gmail.com>wrote:
> On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> <ti...@gmail.com> wrote:
> > Hi Tom,
> >
> > One major difference is that Hadoop allows working with any object
> > that you can serialize, rather than only Strings so it is applicable
> > for use with (e.g) image processing, or rendering to images or PDFs on
> > the output format.
>
> Yes, you noted this on the general@hadoop list. I think your
> suggestion to expand this to support other types is a good one.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Bruce Snyder <br...@gmail.com>.
On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
<ti...@gmail.com> wrote:
> Hi Tom,
>
> One major difference is that Hadoop allows working with any object
> that you can serialize, rather than only Strings so it is applicable
> for use with (e.g) image processing, or rendering to images or PDFs on
> the output format.
Yes, you noted this on the general@hadoop list. I think your
suggestion to expand this to support other types is a good one.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Tim Robertson <ti...@gmail.com>.
Hi Tom,
One major difference is that Hadoop allows working with any object
that you can serialize, rather than only Strings so it is applicable
for use with (e.g) image processing, or rendering to images or PDFs on
the output format.
Cheers
Tim
On Sun, Nov 29, 2009 at 5:36 AM, Bruce Snyder <br...@gmail.com> wrote:
> On Sat, Nov 28, 2009 at 8:55 PM, tom <zs...@gmail.com> wrote:
>> Hi,
>>
>> I know Hadoop is a suite of software service for the cluster
>> computing, including mapreduce implementation, distributed database,
>> distributed file system, etc.
>> the confusion to me is If cloudMapReduce just is focused on Cloud
>> Computing Infrastructure and must be depend on overall distributed
>> service provided and hidden by Cloud Computing Infrastructure?
>>
>> And, cloudMapReduce just can be a alternate implementation of MapReduce?
>> Is it right?
>
> Yes, Cloud MapReduce is an alternate implementation of MapReduce. The
> difference between Cloud MapReduce and Hadoop is that Cloud MapReduce
> depends upon the infrastructure services provided by Amazon (e.g.,
> AMIs, S3, SQS and SimpleDB) instead of providing it's own
> implementation of each.
>
> On a side note, I find it interesting that Amazon is now offering a
> product based on Hadoop named Amazon Elastic MapReduce:
>
> http://aws.amazon.com/elasticmapreduce/
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
Re: what is the major difference between Hadoop and cloudMapReduce?
Posted by Bruce Snyder <br...@gmail.com>.
On Sat, Nov 28, 2009 at 8:55 PM, tom <zs...@gmail.com> wrote:
> Hi,
>
> I know Hadoop is a suite of software service for the cluster
> computing, including mapreduce implementation, distributed database,
> distributed file system, etc.
> the confusion to me is If cloudMapReduce just is focused on Cloud
> Computing Infrastructure and must be depend on overall distributed
> service provided and hidden by Cloud Computing Infrastructure?
>
> And, cloudMapReduce just can be a alternate implementation of MapReduce?
> Is it right?
Yes, Cloud MapReduce is an alternate implementation of MapReduce. The
difference between Cloud MapReduce and Hadoop is that Cloud MapReduce
depends upon the infrastructure services provided by Amazon (e.g.,
AMIs, S3, SQS and SimpleDB) instead of providing it's own
implementation of each.
On a side note, I find it interesting that Amazon is now offering a
product based on Hadoop named Amazon Elastic MapReduce:
http://aws.amazon.com/elasticmapreduce/
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder