You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Robin Anil <ro...@gmail.com> on 2012/05/08 17:11:46 UTC
Making Mahout Leaner
Based on some discussion on the private group about where Mahout is
faltering in the real world, a stream of thought bubbled up - Make Mahout
leaner. i.e push the best stuff we have to the top and prune out algorithms
that are underperforming. The main issue here is that Iterative nature of
many of the algorithms make it inefficient to be implemented on top of
current Hadoop. The summary or the state of the disucssion so far
1) Focus on large scale data(not medium scale) and focus on algorithms that
run at *almost* O(n).
2) Focus on deployability and less on making it an analysis tool for data
competitions.
3) Prune prune prune things that are not being maintained.
The following is one way of looking at Mahout and the state of its
algorithms. Let us know if you would like something to be in the keeper
category.
Keepers
1. Recommenders -- clearly a keeper
2. SGD
3. LDA
4. Some clustering (with upgrades)
5. Math + collections
6. Hadoop Utilities + Integration -- I know it's silly, but things like
sequence file dumper, the iterators, etc. are handy in a number of places.
7. SVD and related
8 RowSimilarity
9. Some of the upfront preprocessing tools (Lucene, Text , etc.)
Unsure:
- Bayes + Random Forest - Seems a shame on bayes, since it gives a
baseline, but I don't know that it actually works and then there's the
whole split personality nature of it (text-based and vector-based)
- Collocations - I'd say keep for now, even if just for selfish reasons
- Minhash - every time I look at it is seems broken and the original author
doesn't respond to requests for explanation.
- Freq. Item Set - Tom's done some work to clean up and I've tried it on
search logs and the results looked OK, but no formal evaluation. I've seen
others say why not just do simpler co-occurrence stuff...
Drop for sure:
1. Watchmaker
2. Unused/poor examples
3. Probably a lot more that escapes me at the moment.
4. PageRank
------
Robin Anil
Re: Making Mahout Leaner
Posted by Ted Dunning <te...@gmail.com>.
To add to this and to save the breath of the participants in the formerly
private discussion, it seems like there is rough consensus about removing
cruft, but there has also been quite a bit of desire to be very sensitive
to the needs of current and planned production users. Somewhat less
intense is a desire to be sensitive to needs of less critical uses.
On Tue, May 8, 2012 at 8:11 AM, Robin Anil <ro...@gmail.com> wrote:
> Based on some discussion on the private group about where Mahout is
> faltering in the real world, a stream of thought bubbled up - Make Mahout
> leaner. i.e push the best stuff we have to the top and prune out algorithms
> that are underperforming. The main issue here is that Iterative nature of
> many of the algorithms make it inefficient to be implemented on top of
> current Hadoop. The summary or the state of the disucssion so far
>
> 1) Focus on large scale data(not medium scale) and focus on algorithms that
> run at *almost* O(n).
> 2) Focus on deployability and less on making it an analysis tool for data
> competitions.
> 3) Prune prune prune things that are not being maintained.
>
> The following is one way of looking at Mahout and the state of its
> algorithms. Let us know if you would like something to be in the keeper
> category.
>
> Keepers
> 1. Recommenders -- clearly a keeper
> 2. SGD
> 3. LDA
> 4. Some clustering (with upgrades)
> 5. Math + collections
> 6. Hadoop Utilities + Integration -- I know it's silly, but things like
> sequence file dumper, the iterators, etc. are handy in a number of places.
> 7. SVD and related
> 8 RowSimilarity
> 9. Some of the upfront preprocessing tools (Lucene, Text , etc.)
>
> Unsure:
>
> - Bayes + Random Forest - Seems a shame on bayes, since it gives a
> baseline, but I don't know that it actually works and then there's the
> whole split personality nature of it (text-based and vector-based)
> - Collocations - I'd say keep for now, even if just for selfish reasons
> - Minhash - every time I look at it is seems broken and the original author
> doesn't respond to requests for explanation.
> - Freq. Item Set - Tom's done some work to clean up and I've tried it on
> search logs and the results looked OK, but no formal evaluation. I've seen
> others say why not just do simpler co-occurrence stuff...
>
> Drop for sure:
>
> 1. Watchmaker
> 2. Unused/poor examples
> 3. Probably a lot more that escapes me at the moment.
> 4. PageRank
> ------
> Robin Anil
>
Re: Making Mahout Leaner
Posted by Jake Mannix <ja...@gmail.com>.
Robin,
Given that our "state of the art" classification currently is the SGD
stuff,
I'd target toward those interfaces (but not worry about the
feature-hashing).
Why do you need new collocation stuff? Can this implementation not
work on raw Vector instances? Let seq2sparse generate the vectors for
you and use that? The tf-idf and tf vectors should both already exist,
whether for unigrams or for ngrams (depending on how you called
seq2sparse), right?
As for integrating with ConfusingMatrix and other eval tools, I have not
too much to offer as advice...
On Thu, May 10, 2012 at 8:39 PM, Robin Anil <ro...@gmail.com> wrote:
> Any directions on what pattern I should follow for the redesign.
> ------
> Robin Anil
>
>
> On Wed, May 9, 2012 at 9:49 AM, Robin Anil <ro...@gmail.com> wrote:
>
> > I believe most of this new NB discussion has been over chat. So here is
> > the state of the NB universe from my view
> >
> > 1) Original NB and CNB code was as follows
> > - Tokenize and find all possible collocations
> > - Compute Tf and Idf for each ngram
> > - Compute Global and per class sums for tf, idf and tf-idf
> > - Dump these counts in SequenceFiles
> > - Load these into memory or Hbase and do compute score for each
> > vector-label combination
> >
> > Issues
> > A) its slow (collocation thought its efficient in its implementation
> (zero
> > memory overhead, using secondary sort), just explodes the learning time)
> > B) Its a memory hog. For really large models you really need Hbase to
> > store counts efficiently for large models. The class has a cache for
> > frequently used words in the language. So the overhead of classification
> is
> > based on the number on infrequent words in the document and the amount of
> > parallel lookups you can do on a Hbase cluster.
> >
> >
> > The new NB and CNB code is as follows:
> > - The redesigned naive bayes doesnt work over words. It assumes input
> > is document vector and computes tf-idf and weights. (This is implemented)
> > - The perclass weight vectors are kept in memory and updated. So the
> > limiting factor here is the number of classes * number of dimensions.
> (This
> > is implemented)
> > - If the vector space is limited using randomized hashing (ted's
> > technique), then you can limit the space. However for (all possible)
> ngrams
> > you will need a large dimension, which makes it unusable. (This is not
> > done).
> > - So one needs to create collocation vectors smartly (This is not
> done).
> > - The implmentation as of now, learns the model, has model
> > serialization and deserialization methods, and an interface for
> classifying
> > using the loaded model. (This is implemented)
> >
> > Issues
> > A) It lacks train and test driver code. Its just has the core
> > implementation
> > B) It is not integrated with the evaluation classes (Confusion Matrix,
> Per
> > label precision/recall)
> > C) We need to port the collocations driver to generate collocations and
> > convert documents to vectors.
> > D) The multilabel classifier is not using any common interface like the
> > logistic regression package.
> >
> > When I checked in the code I didnt have time to pursue this. If someone
> > can recommend, the right approach to fixing this package. Like the right
> > interface to use, How it should behave with rest of the code. It becomes
> > easier for be to jump back on moulding the current implementation.
> >
> > ------
> > Robin Anil
> >
> >
> >
> > On Wed, May 9, 2012 at 5:48 AM, Grant Ingersoll <gsingers@apache.org
> >wrote:
> >
> >>
> >> On May 8, 2012, at 12:43 PM, Jake Mannix wrote:
> >>
> >> > On Tue, May 8, 2012 at 9:31 AM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >> >
> >> >> This is frustrating to consider losing Bayes, but I would consider
> >> keeping
> >> >> it if only to decrease the number of questions on the list about why
> >> the
> >> >> examples from the book don't work.
> >> >>
> >> >
> >> > Could maybe someone just sit down and rewrite it? Naive Bayes is not
> a
> >> > particularly
> >> > difficult thing to implement, even distributed (it's like, word-count,
> >> > basically. Ok,
> >> > maybe it's more like counting collocations, but still!).
> >> >
> >> > It would be pretty silly to not have an NB impl (although I agree that
> >> it's
> >> > even worse
> >> > to have a broken or clunky one).
> >>
> >> I agree. The vector based one is a rewrite, so we probably should just
> >> go from there. Not sure it is broken, but Robin is the primary person
> >> familiar with it and in the past I've pinged the list on the state of it
> >> (and trying to get explanations on certain parts of it) and not gotten
> >> answers.
> >
> >
> >
> >
> >> With all of these Hadoop algorithms, the other thing we really need is
> to
> >> make them programmatically easier to integrate. The Driver mode is not
> too
> >> bad for testing, etc. but it makes it harder to integrate, as others
> have
> >> pointed out.
> >
> >
> >
>
--
-jake
Re: Making Mahout Leaner
Posted by Robin Anil <ro...@gmail.com>.
Any directions on what pattern I should follow for the redesign.
------
Robin Anil
On Wed, May 9, 2012 at 9:49 AM, Robin Anil <ro...@gmail.com> wrote:
> I believe most of this new NB discussion has been over chat. So here is
> the state of the NB universe from my view
>
> 1) Original NB and CNB code was as follows
> - Tokenize and find all possible collocations
> - Compute Tf and Idf for each ngram
> - Compute Global and per class sums for tf, idf and tf-idf
> - Dump these counts in SequenceFiles
> - Load these into memory or Hbase and do compute score for each
> vector-label combination
>
> Issues
> A) its slow (collocation thought its efficient in its implementation (zero
> memory overhead, using secondary sort), just explodes the learning time)
> B) Its a memory hog. For really large models you really need Hbase to
> store counts efficiently for large models. The class has a cache for
> frequently used words in the language. So the overhead of classification is
> based on the number on infrequent words in the document and the amount of
> parallel lookups you can do on a Hbase cluster.
>
>
> The new NB and CNB code is as follows:
> - The redesigned naive bayes doesnt work over words. It assumes input
> is document vector and computes tf-idf and weights. (This is implemented)
> - The perclass weight vectors are kept in memory and updated. So the
> limiting factor here is the number of classes * number of dimensions. (This
> is implemented)
> - If the vector space is limited using randomized hashing (ted's
> technique), then you can limit the space. However for (all possible) ngrams
> you will need a large dimension, which makes it unusable. (This is not
> done).
> - So one needs to create collocation vectors smartly (This is not done).
> - The implmentation as of now, learns the model, has model
> serialization and deserialization methods, and an interface for classifying
> using the loaded model. (This is implemented)
>
> Issues
> A) It lacks train and test driver code. Its just has the core
> implementation
> B) It is not integrated with the evaluation classes (Confusion Matrix, Per
> label precision/recall)
> C) We need to port the collocations driver to generate collocations and
> convert documents to vectors.
> D) The multilabel classifier is not using any common interface like the
> logistic regression package.
>
> When I checked in the code I didnt have time to pursue this. If someone
> can recommend, the right approach to fixing this package. Like the right
> interface to use, How it should behave with rest of the code. It becomes
> easier for be to jump back on moulding the current implementation.
>
> ------
> Robin Anil
>
>
>
> On Wed, May 9, 2012 at 5:48 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On May 8, 2012, at 12:43 PM, Jake Mannix wrote:
>>
>> > On Tue, May 8, 2012 at 9:31 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >
>> >> This is frustrating to consider losing Bayes, but I would consider
>> keeping
>> >> it if only to decrease the number of questions on the list about why
>> the
>> >> examples from the book don't work.
>> >>
>> >
>> > Could maybe someone just sit down and rewrite it? Naive Bayes is not a
>> > particularly
>> > difficult thing to implement, even distributed (it's like, word-count,
>> > basically. Ok,
>> > maybe it's more like counting collocations, but still!).
>> >
>> > It would be pretty silly to not have an NB impl (although I agree that
>> it's
>> > even worse
>> > to have a broken or clunky one).
>>
>> I agree. The vector based one is a rewrite, so we probably should just
>> go from there. Not sure it is broken, but Robin is the primary person
>> familiar with it and in the past I've pinged the list on the state of it
>> (and trying to get explanations on certain parts of it) and not gotten
>> answers.
>
>
>
>
>> With all of these Hadoop algorithms, the other thing we really need is to
>> make them programmatically easier to integrate. The Driver mode is not too
>> bad for testing, etc. but it makes it harder to integrate, as others have
>> pointed out.
>
>
>
Re: Making Mahout Leaner
Posted by Robin Anil <ro...@gmail.com>.
I believe most of this new NB discussion has been over chat. So here is the
state of the NB universe from my view
1) Original NB and CNB code was as follows
- Tokenize and find all possible collocations
- Compute Tf and Idf for each ngram
- Compute Global and per class sums for tf, idf and tf-idf
- Dump these counts in SequenceFiles
- Load these into memory or Hbase and do compute score for each
vector-label combination
Issues
A) its slow (collocation thought its efficient in its implementation (zero
memory overhead, using secondary sort), just explodes the learning time)
B) Its a memory hog. For really large models you really need Hbase to store
counts efficiently for large models. The class has a cache for frequently
used words in the language. So the overhead of classification is based on
the number on infrequent words in the document and the amount of parallel
lookups you can do on a Hbase cluster.
The new NB and CNB code is as follows:
- The redesigned naive bayes doesnt work over words. It assumes input is
document vector and computes tf-idf and weights. (This is implemented)
- The perclass weight vectors are kept in memory and updated. So the
limiting factor here is the number of classes * number of dimensions. (This
is implemented)
- If the vector space is limited using randomized hashing (ted's
technique), then you can limit the space. However for (all possible) ngrams
you will need a large dimension, which makes it unusable. (This is not
done).
- So one needs to create collocation vectors smartly (This is not done).
- The implmentation as of now, learns the model, has model serialization
and deserialization methods, and an interface for classifying using the
loaded model. (This is implemented)
Issues
A) It lacks train and test driver code. Its just has the core implementation
B) It is not integrated with the evaluation classes (Confusion Matrix, Per
label precision/recall)
C) We need to port the collocations driver to generate collocations and
convert documents to vectors.
D) The multilabel classifier is not using any common interface like the
logistic regression package.
When I checked in the code I didnt have time to pursue this. If someone can
recommend, the right approach to fixing this package. Like the right
interface to use, How it should behave with rest of the code. It becomes
easier for be to jump back on moulding the current implementation.
------
Robin Anil
On Wed, May 9, 2012 at 5:48 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On May 8, 2012, at 12:43 PM, Jake Mannix wrote:
>
> > On Tue, May 8, 2012 at 9:31 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> This is frustrating to consider losing Bayes, but I would consider
> keeping
> >> it if only to decrease the number of questions on the list about why the
> >> examples from the book don't work.
> >>
> >
> > Could maybe someone just sit down and rewrite it? Naive Bayes is not a
> > particularly
> > difficult thing to implement, even distributed (it's like, word-count,
> > basically. Ok,
> > maybe it's more like counting collocations, but still!).
> >
> > It would be pretty silly to not have an NB impl (although I agree that
> it's
> > even worse
> > to have a broken or clunky one).
>
> I agree. The vector based one is a rewrite, so we probably should just go
> from there. Not sure it is broken, but Robin is the primary person
> familiar with it and in the past I've pinged the list on the state of it
> (and trying to get explanations on certain parts of it) and not gotten
> answers.
> With all of these Hadoop algorithms, the other thing we really need is to
> make them programmatically easier to integrate. The Driver mode is not too
> bad for testing, etc. but it makes it harder to integrate, as others have
> pointed out.
Re: Making Mahout Leaner
Posted by Grant Ingersoll <gs...@apache.org>.
On May 8, 2012, at 12:43 PM, Jake Mannix wrote:
> On Tue, May 8, 2012 at 9:31 AM, Ted Dunning <te...@gmail.com> wrote:
>
>> This is frustrating to consider losing Bayes, but I would consider keeping
>> it if only to decrease the number of questions on the list about why the
>> examples from the book don't work.
>>
>
> Could maybe someone just sit down and rewrite it? Naive Bayes is not a
> particularly
> difficult thing to implement, even distributed (it's like, word-count,
> basically. Ok,
> maybe it's more like counting collocations, but still!).
>
> It would be pretty silly to not have an NB impl (although I agree that it's
> even worse
> to have a broken or clunky one).
I agree. The vector based one is a rewrite, so we probably should just go from there. Not sure it is broken, but Robin is the primary person familiar with it and in the past I've pinged the list on the state of it (and trying to get explanations on certain parts of it) and not gotten answers.
With all of these Hadoop algorithms, the other thing we really need is to make them programmatically easier to integrate. The Driver mode is not too bad for testing, etc. but it makes it harder to integrate, as others have pointed out.
Re: Making Mahout Leaner
Posted by Jake Mannix <ja...@gmail.com>.
On Tue, May 8, 2012 at 9:31 AM, Ted Dunning <te...@gmail.com> wrote:
> This is frustrating to consider losing Bayes, but I would consider keeping
> it if only to decrease the number of questions on the list about why the
> examples from the book don't work.
>
Could maybe someone just sit down and rewrite it? Naive Bayes is not a
particularly
difficult thing to implement, even distributed (it's like, word-count,
basically. Ok,
maybe it's more like counting collocations, but still!).
It would be pretty silly to not have an NB impl (although I agree that it's
even worse
to have a broken or clunky one).
>
> On Tue, May 8, 2012 at 8:11 AM, Robin Anil <ro...@gmail.com> wrote:
>
> > - Bayes + Random Forest - Seems a shame on bayes, since it gives a
> > baseline, but I don't know that it actually works and then there's the
> > whole split personality nature of it (text-based and vector-based)
> >
>
--
-jake
Re: Making Mahout Leaner
Posted by Ted Dunning <te...@gmail.com>.
This is frustrating to consider losing Bayes, but I would consider keeping
it if only to decrease the number of questions on the list about why the
examples from the book don't work.
On Tue, May 8, 2012 at 8:11 AM, Robin Anil <ro...@gmail.com> wrote:
> - Bayes + Random Forest - Seems a shame on bayes, since it gives a
> baseline, but I don't know that it actually works and then there's the
> whole split personality nature of it (text-based and vector-based)
>