Alternative Naive Bayes Datastore?

[Dean Jones <de...@gmail.com>]

Hello folks,

I've been training a language model using the Naive Bayes classifier
and things have been going pretty well so far (it's been remarkably
straightforward to get everything up and running on EC2 - thanks!).
However, I've now hit a bit of an obstacle in terms of using the
generated model in our application. As far as I can tell, there are
two options - the model is either stored in Hbase or it's all loaded
into memory using the InMemoryBayesDatastore. Unfortunately, Hbase
isn't an option for us, and the models we're generating are too big to
be held in memory (we're doing some feature pruning, but memory
consumption is still an issue). I'm pretty new to Mahout, so I just
wondered whether there was a known workaround to this problem -
perhaps something based on a persistent cache like jdbm?

Thanks,

Dean.

Re: Alternative Naive Bayes Datastore?

[Grant Ingersoll <gs...@apache.org>]

I haven't looked in a while, but I believe the data store is pluggable.  I'm not sure how much is involved in adding your own, but it was one of the design goals. You might start by looking at the two relevant implementations and go from there.  Let us know what works.  Also, if you care to share your implementation, we may be able to incorporate it (pending licensing, etc.)

-Grant

On Sep 14, 2010, at 5:26 AM, Dean Jones wrote:

> Hello folks,
> 
> I've been training a language model using the Naive Bayes classifier
> and things have been going pretty well so far (it's been remarkably
> straightforward to get everything up and running on EC2 - thanks!).
> However, I've now hit a bit of an obstacle in terms of using the
> generated model in our application. As far as I can tell, there are
> two options - the model is either stored in Hbase or it's all loaded
> into memory using the InMemoryBayesDatastore. Unfortunately, Hbase
> isn't an option for us, and the models we're generating are too big to
> be held in memory (we're doing some feature pruning, but memory
> consumption is still an issue). I'm pretty new to Mahout, so I just
> wondered whether there was a known workaround to this problem -
> perhaps something based on a persistent cache like jdbm?
> 
> Thanks,
> 
> Dean.

Re: Alternative Naive Bayes Datastore?

[Ted Dunning <te...@gmail.com>]

The serialization involved in that comment is likely to be GSON based
serialization.

See org.apache.mahout.classifier.sgd.ModelSerializer for uses of this.

On Tue, Sep 14, 2010 at 8:38 AM, Dean Jones <de...@gmail.com> wrote:

> HI Grant, Drew,
>
> Thanks for the responses. I had pretty much concluded that I would
> need to roll my own datastore, but thanks for the confirmation. I'm
> basing the implementation on the InMemoryBayesDatastore, but have hit
> a little bit of a snag because I'm trying to implement a jdbm-backed
> version of o.a.m.math.Matrix using
> o.a.m.math.RandomAccessSparseVector, but this class is not
> Serializable; it does have a default constructor with the comment "For
> serialization purposes only", which suggests that perhaps that
> intention was to allow its subclasses to be Serializable. Any thoughts
> on what the intention here was? Currently I've patched it to implement
> Serializable.
>
> Dean.
>
> On 14 September 2010 14:22, Drew Farris <dr...@apache.org> wrote:
> > Hi Dean,
> >
> > InMemory and HBase are the only options present currently. It
> > shouldn't be too difficult to implement a new storage back-end. You
> > would need to consider the issue of both loading the data into your
> > persistence framework and reading it back out.
> >
>

Re: Alternative Naive Bayes Datastore?

[Ted Dunning <te...@gmail.com>]

The only significant advantages that I am aware of are that our linear
algebra code have in themselves are

- alternative sparse vector formats that can result in substantial speedups
for certain operations

- solid interfaces with hadoop in the form of our VectorWritable

- we still have a flexible compatibility policy so that if there is
something you need, we can add it in less than a year.  We originally wanted
to use commons math, but they couldn't add the capabilities we needed in a
reasonable time frame.

Of course, another advantage is that the Mahout machine learning code uses
them so if you use them, you can interface more easily with our stuff.  But
that isn't really an intrinsic advantage.

On Fri, Sep 17, 2010 at 1:58 AM, Dean Jones <de...@gmail.com> wrote:

> I'd be interested though in any advantages that the Mahout linear
> algebra library has over the commons-math library.
>

Re: Alternative Naive Bayes Datastore?

[Dean Jones <de...@gmail.com>]

Hi Ted,

>
> I understood that.  What I was hoping for was feedback on how our sparse
> vectors could be made better based on comparison with your internal library.
>

The only "advantage" our internal library has that I'm aware of is
that the vector and matrix classes implement Serializable, which
allows us to more easily interface with other libraries that require
this. As a matter of fact, over the last year or so, we've preferred
to use the commons-math library and considered ours largely
deprecated. IIRC, the commons-math classes also implement
Serializable, so that might also be an option for us here.

> I was also suggesting that in other areas, interface with Mahout might be
> made very easy without forcing you to migrate your current vector package
> very much.  That is a particular
> interest since this is the first potential use case for a user masquerading
> their vector package as compatible with Mahout vectors.
>

Ah, I see. As I say, nowadays we prefer to use the commons-math
library, so we're unlikely to do much work on our internal library.
I'd be interested though in any advantages that the Mahout linear
algebra library has over the commons-math library.

Dean.

Re: Alternative Naive Bayes Datastore?

[Ted Dunning <te...@gmail.com>]

On Thu, Sep 16, 2010 at 3:06 PM, Dean Jones <de...@gmail.com> wrote:

> Hi Ted,
>
> On 16 September 2010 20:48, Ted Dunning <te...@gmail.com> wrote:
> > Do those implementations have something Mahout doesn't?
> >
> > It isn't that I expect that you could switch, just looking to make ours
> as
> > good as anything reasonable.
> >
>
> I think there may be some confusion, apologies if I wasn't clear. I
> didn't mean to suggest that we would contribute our sparse vector code
> (for a start, I know that Mahout has it's own linear algebra library
> and doesn't need ours) but that we'd happily contribute the
> jdbm-backed implementation of the Mahout Bayes Datastore that we have,
> if (i) you'd like it, and (ii) we can find a resolution to the problem
> of storing Mahout vectors in jdbm.
>

I understood that.  What I was hoping for was feedback on how our sparse
vectors could be made better based on comparison with your internal library.

I was also suggesting that in other areas, interface with Mahout might be
made very easy without forcing you to migrate your current vector package
very much.  That is a particular
interest since this is the first potential use case for a user masquerading
their vector package
as compatible with Mahout vectors.

Re: Alternative Naive Bayes Datastore?

[Dean Jones <de...@gmail.com>]

Hi Ted,

On 16 September 2010 20:48, Ted Dunning <te...@gmail.com> wrote:
> Do those implementations have something Mahout doesn't?
>
> It isn't that I expect that you could switch, just looking to make ours as
> good as anything reasonable.
>

I think there may be some confusion, apologies if I wasn't clear. I
didn't mean to suggest that we would contribute our sparse vector code
(for a start, I know that Mahout has it's own linear algebra library
and doesn't need ours) but that we'd happily contribute the
jdbm-backed implementation of the Mahout Bayes Datastore that we have,
if (i) you'd like it, and (ii) we can find a resolution to the problem
of storing Mahout vectors in jdbm.

Best wishes,

Dean.

Re: Alternative Naive Bayes Datastore?

[Ted Dunning <te...@gmail.com>]

Do those implementations have something Mahout doesn't?

It isn't that I expect that you could switch, just looking to make ours as
good as anything reasonable.

If you find Mahout code useful, you should be able to implement Mahout's
Vector interface and use your vectors interchangeably.  One of the goals of
our original implementation was to allow new implementations in ways that
the original Colt code really didn't allow.

On Thu, Sep 16, 2010 at 5:07 AM, Dean Jones <de...@gmail.com> wrote:

> For me, we have our own sparse vector implementations, so I can switch
> to using those, but that would obviously impact my ability to
> contribute this work back to mahout, which is something I'd like to do
> (and have got clearance from my employer for).
>

Re: Alternative Naive Bayes Datastore?

[Dean Jones <de...@gmail.com>]

Hi Drew,

On 15 September 2010 13:58, Drew Farris <dr...@gmail.com> wrote:
> Hi Dean,
>
> Does jdbm only support java-based serialization? From my experience
> I've seen that java's serialization is generally an order of magnitude
> slower and less space efficient than the equivalent hand-rolled
> serialization such as you'd find in implementations of the Writable
> class. That is precisely why you won't see Serializable used much in
> Mahout. Perhaps you could use RandomAccessSparseVector combined with
> VectorWritable to read/write to/from a byte array backed
> DataOutput/DataInput stream?
>

As far as I can tell, the jdbm HTree and BTree stores do only support
java serialization. I've patched RandomAccessSparseVector locally to
implement Serializable, and this is now working for me. I absolutely
agree that the native java serialization is not the best, but it is
widely used by popular persistence and ipc mechanisms, which may be
something you want to consider.

For me, we have our own sparse vector implementations, so I can switch
to using those, but that would obviously impact my ability to
contribute this work back to mahout, which is something I'd like to do
(and have got clearance from my employer for).

Hmm, maybe I should create an issue on the mahout jira, attach a
patch, and we could take it from there?

Dean.

Re: Alternative Naive Bayes Datastore?

[Drew Farris <dr...@gmail.com>]

Hi Dean,

Does jdbm only support java-based serialization? From my experience
I've seen that java's serialization is generally an order of magnitude
slower and less space efficient than the equivalent hand-rolled
serialization such as you'd find in implementations of the Writable
class. That is precisely why you won't see Serializable used much in
Mahout. Perhaps you could use RandomAccessSparseVector combined with
VectorWritable to read/write to/from a byte array backed
DataOutput/DataInput stream?

I suspect the HBase interactions might be a useful starting point in
terms of observing how the matrices are broken down into chunks of
data loaded into/read from a persistent store.

I'm definitely interested in how things proceed for you, especially if
there are ways the existing code can be improved for easier extension
in the future.

Drew

On Tue, Sep 14, 2010 at 11:38 AM, Dean Jones <de...@gmail.com> wrote:
> HI Grant, Drew,
>
> Thanks for the responses. I had pretty much concluded that I would
> need to roll my own datastore, but thanks for the confirmation. I'm
> basing the implementation on the InMemoryBayesDatastore, but have hit
> a little bit of a snag because I'm trying to implement a jdbm-backed
> version of o.a.m.math.Matrix using
> o.a.m.math.RandomAccessSparseVector, but this class is not
> Serializable; it does have a default constructor with the comment "For
> serialization purposes only", which suggests that perhaps that
> intention was to allow its subclasses to be Serializable. Any thoughts
> on what the intention here was? Currently I've patched it to implement
> Serializable.
>
> Dean.
>
> On 14 September 2010 14:22, Drew Farris <dr...@apache.org> wrote:
>> Hi Dean,
>>
>> InMemory and HBase are the only options present currently. It
>> shouldn't be too difficult to implement a new storage back-end. You
>> would need to consider the issue of both loading the data into your
>> persistence framework and reading it back out.
>>
>

Re: Alternative Naive Bayes Datastore?

[Dean Jones <de...@gmail.com>]

HI Grant, Drew,

Thanks for the responses. I had pretty much concluded that I would
need to roll my own datastore, but thanks for the confirmation. I'm
basing the implementation on the InMemoryBayesDatastore, but have hit
a little bit of a snag because I'm trying to implement a jdbm-backed
version of o.a.m.math.Matrix using
o.a.m.math.RandomAccessSparseVector, but this class is not
Serializable; it does have a default constructor with the comment "For
serialization purposes only", which suggests that perhaps that
intention was to allow its subclasses to be Serializable. Any thoughts
on what the intention here was? Currently I've patched it to implement
Serializable.

Dean.

On 14 September 2010 14:22, Drew Farris <dr...@apache.org> wrote:
> Hi Dean,
>
> InMemory and HBase are the only options present currently. It
> shouldn't be too difficult to implement a new storage back-end. You
> would need to consider the issue of both loading the data into your
> persistence framework and reading it back out.
>

Re: Alternative Naive Bayes Datastore?

[Drew Farris <dr...@apache.org>]

Hi Dean,

InMemory and HBase are the only options present currently. It
shouldn't be too difficult to implement a new storage back-end. You
would need to consider the issue of both loading the data into your
persistence framework and reading it back out.

When it comes to writing a model to storage you have two options. One
option would be implementing a translation layer from the hdfs format
to your storage mechanism of choice. Another option is to look at how
the various reducers involved in the Bayes training process write data
directly to HBase and build on this.

Reading the model is perhaps more straightforward. It would not be
difficult to implement your own datastore by providing an alternate
implementation of the o.a.m.classifier.bayes.interfaces.Datastore
interface. You could follow the general pattern established in the
HBaseBayesDatastore regardless of the persistence mechanism.

HTH,

Drew

On Tue, Sep 14, 2010 at 5:26 AM, Dean Jones <de...@gmail.com> wrote:
> Hello folks,
>
> I've been training a language model using the Naive Bayes classifier
> and things have been going pretty well so far (it's been remarkably
> straightforward to get everything up and running on EC2 - thanks!).
> However, I've now hit a bit of an obstacle in terms of using the
> generated model in our application. As far as I can tell, there are
> two options - the model is either stored in Hbase or it's all loaded
> into memory using the InMemoryBayesDatastore. Unfortunately, Hbase
> isn't an option for us, and the models we're generating are too big to
> be held in memory (we're doing some feature pruning, but memory
> consumption is still an issue). I'm pretty new to Mahout, so I just
> wondered whether there was a known workaround to this problem -
> perhaps something based on a persistent cache like jdbm?
>
> Thanks,
>
> Dean.
>