You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stefan Groschupf <sg...@101tec.com> on 2006/10/27 01:13:43 UTC
Re: Statistical clustering MapReduce example?
Hi David,
a student once wrote a paper about map reduce and clustering in my
company during a internal ship.
I will send it to you off list since the list does not support
attachments.
However if someone wants to have a copy as well, let me know.
Cheers,
Stefan
Am 23.10.2006 um 22:41 schrieb David Pollak:
> Howdy,
>
> I'm looking to cluster documents by word frequency (and maybe
> position). Does anyone know of MapReduce examples that demonstrate
> statistical clustering?
>
> Thanks,
>
> David
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
search tech for web 2.1
Menlo Park, California
http://www.101tec.com
Re: Statistical clustering MapReduce example?
Posted by Dennis Kubes <nu...@dragonflymc.com>.
Stefan Groschupf wrote:
> Hi David,
>
> a student once wrote a paper about map reduce and clustering in my
> company during a internal ship.
> I will send it to you off list since the list does not support
> attachments.
> However if someone wants to have a copy as well, let me know.
>
> Cheers,
> Stefan
>
> Am 23.10.2006 um 22:41 schrieb David Pollak:
>
>> Howdy,
>>
>> I'm looking to cluster documents by word frequency (and maybe
>> position). Does anyone know of MapReduce examples that demonstrate
>> statistical clustering?
>>
>> Thanks,
>>
>> David
>>
>>
>>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> search tech for web 2.1
> Menlo Park, California
> http://www.101tec.com
>
>
>
>
Re: Statistical clustering MapReduce example?
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi Albert, Hi All,
yes it works pretty well. Back than we used weta as map reduce
implementation, a system we wrote since hadoop was part of nutch and
difficult to use stand alone.
With hadoop the performance should be even better also the
implementation should be much easier.
We do a lot of text mining with map reduce and it works/scale great.
It would be nice to have a big table implementation for some tasks
that need to share a fast "memory" but I'm busy with other stuff
these days to help working on that as well.
We worked around this problem by having a set of boxes that have a
big RAM and hold a simple HashMap in memory.
We use a hadoop partitioner to decide which key value tuple we push
into which hashmap server.
Stefan
Am 27.10.2006 um 21:31 schrieb Albert Chern:
> Thank you for the paper Stefan. I was surprised at how well the
> K-Means algorithm fits into MapReduce. I actually never thought of
> writing the output back to the FS to test for convergence.
>
> On 10/27/06, David Pollak <dp...@athena.com> wrote:
>> Stefan,
>>
>> THis is most excellent stuff! Thanks for sending me the paper!
>>
>> David
>>
>> On Oct 26, 2006, at 4:13 PM, Stefan Groschupf wrote:
>>
>> > Hi David,
>> >
>> > a student once wrote a paper about map reduce and clustering in my
>> > company during a internal ship.
>> > I will send it to you off list since the list does not support
>> > attachments.
>> > However if someone wants to have a copy as well, let me know.
>> >
>> > Cheers,
>> > Stefan
>> >
>> > Am 23.10.2006 um 22:41 schrieb David Pollak:
>> >
>> >> Howdy,
>> >>
>> >> I'm looking to cluster documents by word frequency (and maybe
>> >> position). Does anyone know of MapReduce examples that
>> >> demonstrate statistical clustering?
>> >>
>> >> Thanks,
>> >>
>> >> David
>> >>
>> >>
>> >>
>> >
>> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > 101tec Inc.
>> > search tech for web 2.1
>> > Menlo Park, California
>> > http://www.101tec.com
>> >
>> >
>> >
>>
>>
>>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
search tech for web 2.1
Menlo Park, California
http://www.101tec.com
Re: Statistical clustering MapReduce example?
Posted by David Pollak <dp...@athena.com>.
Folks,
I've posted the paper at https://www.lostlake.org/MapReduceKMeans.pdf
Thanks,
David
On Oct 27, 2006, at 12:31 PM, Albert Chern wrote:
> Thank you for the paper Stefan. I was surprised at how well the
> K-Means algorithm fits into MapReduce. I actually never thought of
> writing the output back to the FS to test for convergence.
>
> On 10/27/06, David Pollak <dp...@athena.com> wrote:
>> Stefan,
>>
>> THis is most excellent stuff! Thanks for sending me the paper!
>>
>> David
>>
>> On Oct 26, 2006, at 4:13 PM, Stefan Groschupf wrote:
>>
>> > Hi David,
>> >
>> > a student once wrote a paper about map reduce and clustering in my
>> > company during a internal ship.
>> > I will send it to you off list since the list does not support
>> > attachments.
>> > However if someone wants to have a copy as well, let me know.
>> >
>> > Cheers,
>> > Stefan
>> >
>> > Am 23.10.2006 um 22:41 schrieb David Pollak:
>> >
>> >> Howdy,
>> >>
>> >> I'm looking to cluster documents by word frequency (and maybe
>> >> position). Does anyone know of MapReduce examples that
>> >> demonstrate statistical clustering?
>> >>
>> >> Thanks,
>> >>
>> >> David
>> >>
>> >>
>> >>
>> >
>> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > 101tec Inc.
>> > search tech for web 2.1
>> > Menlo Park, California
>> > http://www.101tec.com
>> >
>> >
>> >
>>
>>
>>
Re: Re: Statistical clustering MapReduce example?
Posted by Albert Chern <al...@gmail.com>.
Thank you for the paper Stefan. I was surprised at how well the
K-Means algorithm fits into MapReduce. I actually never thought of
writing the output back to the FS to test for convergence.
On 10/27/06, David Pollak <dp...@athena.com> wrote:
> Stefan,
>
> THis is most excellent stuff! Thanks for sending me the paper!
>
> David
>
> On Oct 26, 2006, at 4:13 PM, Stefan Groschupf wrote:
>
> > Hi David,
> >
> > a student once wrote a paper about map reduce and clustering in my
> > company during a internal ship.
> > I will send it to you off list since the list does not support
> > attachments.
> > However if someone wants to have a copy as well, let me know.
> >
> > Cheers,
> > Stefan
> >
> > Am 23.10.2006 um 22:41 schrieb David Pollak:
> >
> >> Howdy,
> >>
> >> I'm looking to cluster documents by word frequency (and maybe
> >> position). Does anyone know of MapReduce examples that
> >> demonstrate statistical clustering?
> >>
> >> Thanks,
> >>
> >> David
> >>
> >>
> >>
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 101tec Inc.
> > search tech for web 2.1
> > Menlo Park, California
> > http://www.101tec.com
> >
> >
> >
>
>
>
Re: Statistical clustering MapReduce example?
Posted by David Pollak <dp...@athena.com>.
Stefan,
THis is most excellent stuff! Thanks for sending me the paper!
David
On Oct 26, 2006, at 4:13 PM, Stefan Groschupf wrote:
> Hi David,
>
> a student once wrote a paper about map reduce and clustering in my
> company during a internal ship.
> I will send it to you off list since the list does not support
> attachments.
> However if someone wants to have a copy as well, let me know.
>
> Cheers,
> Stefan
>
> Am 23.10.2006 um 22:41 schrieb David Pollak:
>
>> Howdy,
>>
>> I'm looking to cluster documents by word frequency (and maybe
>> position). Does anyone know of MapReduce examples that
>> demonstrate statistical clustering?
>>
>> Thanks,
>>
>> David
>>
>>
>>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> search tech for web 2.1
> Menlo Park, California
> http://www.101tec.com
>
>
>
Re: Statistical clustering MapReduce example?
Posted by Michael Granitzer <mg...@gmail.com>.
Hi Stefan,
i am also very interested in this topic and would like to take your offer.
Can you send me a copy of the paper as well?
Thanks a lot, cheers
Michael
2006/10/27, Stefan Groschupf <sg...@101tec.com>:
>
> Hi David,
>
> a student once wrote a paper about map reduce and clustering in my
> company during a internal ship.
> I will send it to you off list since the list does not support
> attachments.
> However if someone wants to have a copy as well, let me know.
>
> Cheers,
> Stefan
>
> Am 23.10.2006 um 22:41 schrieb David Pollak:
>
> > Howdy,
> >
> > I'm looking to cluster documents by word frequency (and maybe
> > position). Does anyone know of MapReduce examples that demonstrate
> > statistical clustering?
> >
> > Thanks,
> >
> > David
> >
> >
> >
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> search tech for web 2.1
> Menlo Park, California
> http://www.101tec.com
>
>
>
>
>
Re: Statistical clustering MapReduce example?
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi together,
since so many people asked me, I uploaded the paper to my blog.
http://www.find23.net/MapReduceKMeans.pdf
Cheers,
Stefan
Am 27.10.2006 um 03:33 schrieb Dennis Kubes:
> I would like a copy too please.
>
> Dennis
>
> Stefan Groschupf wrote:
>> Hi David,
>>
>> a student once wrote a paper about map reduce and clustering in my
>> company during a internal ship.
>> I will send it to you off list since the list does not support
>> attachments.
>> However if someone wants to have a copy as well, let me know.
>>
>> Cheers,
>> Stefan
>>
>> Am 23.10.2006 um 22:41 schrieb David Pollak:
>>
>>> Howdy,
>>>
>>> I'm looking to cluster documents by word frequency (and maybe
>>> position). Does anyone know of MapReduce examples that
>>> demonstrate statistical clustering?
>>>
>>> Thanks,
>>>
>>> David
>>>
>>>
>>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 101tec Inc.
>> search tech for web 2.1
>> Menlo Park, California
>> http://www.101tec.com
>>
>>
>>
>>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
search tech for web 2.1
Menlo Park, California
http://www.101tec.com