Mahout Clustering - integration with HBase

[Michael Kurze <mk...@mozilla.com>]

Hello Mahout users,

First, allow me to share some background about the project I am working on: 
to process Firefox user feedback (and eventually feedback submitted with 
crash reports as well, and probably other sources), at Mozilla we are 
looking into developing an online clustering service based on Mahout, with 
HBase as the data store. I wrote more about it on Mozilla’s blog of data:

http://blog.mozilla.com/data/2011/03/08/scalable-text-clustering-for-the-web/

First, we want to start with canopy+k-means to build our proof-of-concept, 
but we are looking into more advanced techniques as the next step.

Browsing through "Mahout in Action" I saw a nice example that gets you 
started with updateable clustering (Listing 9.4 in the 1/19 snapshot). My 
question now is: are there any ideas or even best practices on reading and 
writing documents, vectors and clusters directly from and to HBase rather 
than the file system (HDFS).

Of course I don’t expect some step-by-step solution, but I would appreciate 
pointers on where I would hook in with Mahout to do that, and if it can 
(and should) be done without modifying Mahout source code. Also, I 
appreciate pointers to any previous threads dealing with this topic.

Last, Isabel Drost kindly informed me that something similar has been done 
at Adobe (and that Cosmin Lehene presented about it at Devoxx, [0]), and I’ll
try to get in touch with them as well.

Thanks, Michael



[0] http://www.devoxx.com/display/Devoxx2K10/Big+Data+with+HBase+and+Hadoop+at+Adobe

Re: Mahout Clustering - integration with HBase

[Sebastian Schelter <ss...@apache.org>]

It would be very nice to see the Mozilla Foundation on our "powered by" 
page by the way :)

--sebastian

On 10.03.2011 19:43, Michael Kurze wrote:
> Hello Mahout users,
>
> First, allow me to share some background about the project I am working on:
> to process Firefox user feedback (and eventually feedback submitted with
> crash reports as well, and probably other sources), at Mozilla we are
> looking into developing an online clustering service based on Mahout, with
> HBase as the data store. I wrote more about it on Mozilla’s blog of data:
>
> http://blog.mozilla.com/data/2011/03/08/scalable-text-clustering-for-the-web/
>
> First, we want to start with canopy+k-means to build our proof-of-concept,
> but we are looking into more advanced techniques as the next step.
>
> Browsing through "Mahout in Action" I saw a nice example that gets you
> started with updateable clustering (Listing 9.4 in the 1/19 snapshot). My
> question now is: are there any ideas or even best practices on reading and
> writing documents, vectors and clusters directly from and to HBase rather
> than the file system (HDFS).
>
> Of course I don’t expect some step-by-step solution, but I would appreciate
> pointers on where I would hook in with Mahout to do that, and if it can
> (and should) be done without modifying Mahout source code. Also, I
> appreciate pointers to any previous threads dealing with this topic.
>
> Last, Isabel Drost kindly informed me that something similar has been done
> at Adobe (and that Cosmin Lehene presented about it at Devoxx, [0]), and I’ll
> try to get in touch with them as well.
>
> Thanks, Michael
>
>
>
> [0] http://www.devoxx.com/display/Devoxx2K10/Big+Data+with+HBase+and+Hadoop+at+Adobe
>

Re: Mahout Clustering - integration with HBase

[Ted Dunning <te...@gmail.com>]

On Fri, Mar 11, 2011 at 4:06 AM, Michael Kurze <mk...@mozilla.com> wrote:

> > Updates can be pretty fast as well (I have seen nearly 200K small updates
> per second on half a dozen nodes).
>
> Yes, our table schema is optimized for scanning, that is among the reasons
> we picked HBase vs. simple key/value stores. A second reason would be that
> we also have a large number of small collections to cluster (user feedback
> about individual websites being broken in Firefox, each 10s to 1000s of
> small docs). Since that is more close to random access than streaming, HBase
> seems like a good idea. For those collections, ideally we would talk to
> Mahout in a way so that it does not hit HDFS at all.
>

That should be quite doable.

Your key design will be critical.  The figure of merit is, of course, useful
rows scanned per second.  That will be affected positively by including more
regions in the scan and negatively by having lower density of interesting
rows in the regions being scanned.  If you could cluster all collections in
the same map-reduce job, you have the ideal situation because essentially
all rows of your dataset are live and useful.  At that point, I would
consider arranging the key so that a single collection is contiguous unless
you have to load a single collection at a time at high speed in which case,
I would arrange the key to spread collections.

You will need to adapt the current clustering code so that it understands
that multiple clusterings are going on simultaneously and so that it can
vectorize your rows as you read them.  Vectorization may be expensive enough
that you would rather store feature vectors in hbase.

The two phases of k-means iterations (assignment to cluster, computation of
cluster centroids) can easily be expressed in terms of map-reduce and you
have strong example code already.  I am not quite sure the current state of
play relative to how easily you can serialize cluster descriptions, but that
would be key.

Your numbers on updates are very encouraging, that’s about the cluster size
> we’re starting with!


Contact me off-list for details about how to do this.

Re: Mahout Clustering - integration with HBase

[Michael Kurze <mk...@mozilla.com>]

Robin and Ted, thanks for your feedback!

On Mar 11, 2011, at 12:20 AM, Ted Dunning wrote:

> Actually hbase is really good at scanning.
> 
> Updates can be pretty fast as well (I have seen nearly 200K small updates per second on half a dozen nodes).   

Yes, our table schema is optimized for scanning, that is among the reasons we picked HBase vs. simple key/value stores. A second reason would be that we also have a large number of small collections to cluster (user feedback about individual websites being broken in Firefox, each 10s to 1000s of small docs). Since that is more close to random access than streaming, HBase seems like a good idea. For those collections, ideally we would talk to Mahout in a way so that it does not hit HDFS at all.

Your numbers on updates are very encouraging, that’s about the cluster size we’re starting with!

In general, the best (still generic) interface to Mahout for us would probably (roughly) be one that accepts iterables over vectors and returns an iterator of (centroid, cluster). The iterable passed in by clients could open an hbase scanner for each iteration (or whatever an app needs to do to get data). From what I’ve seen in the KMeans source that might actually not be too far off from what’s already there. The result iterator could then either be fed from an HDFS file or (generator style) from the final stage of the algorithm as it is running.

To get things started, we’ll probably do the roundtrip to HDFS like outlined in the MIA example. That’ll also give us a base of comparison to evaluate future enhancements.


> You can achieve even higher write rates if you create hfiles off-line and then convert them into tables administratively instead of writing to live tables.

For now we plan to load the generated clusters online, since we will also have lots of smaller collections that we want to update close to immediately. Loading tables off-line is still an interesting suggestion, especially in a situation where we might change overall clustering parameters and need to recompute everything.


> 
> On Thu, Mar 10, 2011 at 1:29 PM, Robin Anil <ro...@gmail.com> wrote:
> > Browsing through "Mahout in Action" I saw a nice example that gets you
> > started with updateable clustering (Listing 9.4 in the 1/19 snapshot). My
> > question now is: are there any ideas or even best practices on reading and
> > writing documents, vectors and clusters directly from and to HBase rather
> > than the file system (HDFS).
> >
> 
> Clustering is a very expensive process and hdfs is designed to read data
> sequentially just the way kmeans likes it.
> So given that reading and writing files of HBASE is going to be very
> expensive for repeated iterations, as compared to one time dump, cluster,
> write cluster info back into hbase (maybe into a different column family)

Re: Mahout Clustering - integration with HBase

[Ted Dunning <te...@gmail.com>]

Actually hbase is really good at scanning.

Updates can be pretty fast as well (I have seen nearly 200K small updates
per second on half a dozen nodes).

You can achieve even higher write rates if you create hfiles off-line and
then convert them into tables administratively instead of writing to live
tables.

On Thu, Mar 10, 2011 at 1:29 PM, Robin Anil <ro...@gmail.com> wrote:

> > Browsing through "Mahout in Action" I saw a nice example that gets you
> > started with updateable clustering (Listing 9.4 in the 1/19 snapshot). My
> > question now is: are there any ideas or even best practices on reading
> and
> > writing documents, vectors and clusters directly from and to HBase rather
> > than the file system (HDFS).
> >
>
> Clustering is a very expensive process and hdfs is designed to read data
> sequentially just the way kmeans likes it.
> So given that reading and writing files of HBASE is going to be very
> expensive for repeated iterations, as compared to one time dump, cluster,
> write cluster info back into hbase (maybe into a different column family)

Re: Mahout Clustering - integration with HBase

[Robin Anil <ro...@gmail.com>]

Thanks for getting in touch Michael, a few suggestions in line.

On Fri, Mar 11, 2011 at 12:13 AM, Michael Kurze <mk...@mozilla.com> wrote:

> Hello Mahout users,
>
> First, allow me to share some background about the project I am working on:
> to process Firefox user feedback (and eventually feedback submitted with
> crash reports as well, and probably other sources), at Mozilla we are
> looking into developing an online clustering service based on Mahout, with
> HBase as the data store. I wrote more about it on Mozilla’s blog of data:
>
>
> http://blog.mozilla.com/data/2011/03/08/scalable-text-clustering-for-the-web/
>
> First, we want to start with canopy+k-means to build our proof-of-concept,
> but we are looking into more advanced techniques as the next step.
>
> Browsing through "Mahout in Action" I saw a nice example that gets you
> started with updateable clustering (Listing 9.4 in the 1/19 snapshot). My
> question now is: are there any ideas or even best practices on reading and
> writing documents, vectors and clusters directly from and to HBase rather
> than the file system (HDFS).
>

Clustering is a very expensive process and hdfs is designed to read data
sequentially just the way kmeans likes it.
So given that reading and writing files of HBASE is going to be very
expensive for repeated iterations, as compared to one time dump, cluster,
write cluster info back into hbase (maybe into a different column family)


> Of course I don’t expect some step-by-step solution, but I would appreciate
> pointers on where I would hook in with Mahout to do that, and if it can
> (and should) be done without modifying Mahout source code. Also, I
> appreciate pointers to any previous threads dealing with this topic.
>
> I dont recall anything about clustering over HBASE, but I had tried
classification over HBASE, issue MAHOUT-124 on JIRA, in which I would train
by reading off sequence files from HDFS and writing the model into HBASE.
Then using HBASE like a large matrix, the classifier would get weights from
various columns. But here I had to implement caching to  speed up
classification. Cache would get filled by most frequent words in the corpus.
So for every document the classifier need to fetch only the delta words from
the HBASE.



> Last, Isabel Drost kindly informed me that something similar has been done
> at Adobe (and that Cosmin Lehene presented about it at Devoxx, [0]), and
> I’ll
> try to get in touch with them as well.
>
> Thanks, Michael
>
>
>
> [0]
> http://www.devoxx.com/display/Devoxx2K10/Big+Data+with+HBase+and+Hadoop+at+Adobe
>
>