lsi

[Sebastian Schelter <ss...@apache.org>]

Is there some documentation/tutorial available on how to build a LSI
pipeline with mahout and lucene?

--sebastian

Re: lsi

[Ted Dunning <te...@gmail.com>]

It is possible to index/vectorize new documents in an existing projection.
 Building the projection is pretty much a from-scratch operation.
 Rebuilding the projection can be done pretty infrequently.

On Thu, Nov 17, 2011 at 1:47 PM, Grant Ingersoll <gs...@apache.org>wrote:

> I've never implemented LSI.  Is there a way to incrementally build the
> model (by simply indexing documents) or is it something that one only runs
> after the fact once one has built up the much bigger matrix?  If it's the
> former, I bet it wouldn't be that hard to just implement the appropriate
> new codecs and similarity, assuming Lucene trunk.  If it's the latter, then
> Ted's comment about pushing back into Lucene gets a bit hairier.  Still, I
> wonder if the Codecs/Similarity could help here, too.
>
> What's a typical workflow look like for building all of this?
>
> On Nov 13, 2011, at 3:58 PM, Ted Dunning wrote:
>
> > Essentially not.
> >
> > And I would worry about how to push the LSI vectors back into lucene in a
> > coherent and usable way.
> >
> > On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org>
> wrote:
> >
> >> Is there some documentation/tutorial available on how to build a LSI
> >> pipeline with mahout and lucene?
> >>
> >> --sebastian
> >>
>
>
>

Re: lsi

[Dmitriy Lyubimov <dl...@gmail.com>]

PS the danger of using an overly specific corpus is that training may
not be able to learn polisemy very well unless it sees other documents
with examples of use of the industry jargon words that may also mean
something else. But you definitely want to include documents that do
have words pertinent to the domain, as a first requirement. Actually,
i think you may want to have a balanced corpus of all ways of use of
the words that are specific to your domain. It's kind of hard to find
the right balance, but you can help with the skew towards more
relevant things.

On Thu, Nov 17, 2011 at 4:49 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> The only way to build model incrementally is to do a 'fold in' of new
> observations, that i know.
>
> However, folding in (which is just a multiplication of a new vector
> over the matrices as Ted explained somewhere else) is just a
> projection into already trained space of factors, but not a repetition
> of a training, which is what the actual SVD is.
>
> This may work in more ways than just for the sake of incremental
> compilation. E.g. what i kind of doing: I am working on a particular
> industry domain, so i look for a large corpus of documents pertinent
> to that domain. That's what i run LSA on.
>
> Then i actually throw away the entire document matrix and build a 100%
> fold-in document index based on the term and singular value matrix.
>
> The reason for this is that often the actual corpus of documents that
> you are working with and need proximity comparisons to is not that big
> at all to provide true picture of the specific domain you want to fit
> the stuff into. So generic workflow that i figured in my case  is to
> take a really big corpus relevant to your business, fit term
> dictionary to it using LSA, and then use that dictionary to fold-in
> new documents, with assumption that that big corpus is more relevant
> and much bigger to what you want to focus on, than those documents you
> actually want to retrieve and compare.
>
> Then you may have one fairly infrequent training and a fairly simple
> fold-in procedures.
>
> On Thu, Nov 17, 2011 at 1:47 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> I've never implemented LSI.  Is there a way to incrementally build the model (by simply indexing documents) or is it something that one only runs after the fact once one has built up the much bigger matrix?  If it's the former, I bet it wouldn't be that hard to just implement the appropriate new codecs and similarity, assuming Lucene trunk.  If it's the latter, then Ted's comment about pushing back into Lucene gets a bit hairier.  Still, I wonder if the Codecs/Similarity could help here, too.
>>
>> What's a typical workflow look like for building all of this?
>>
>> On Nov 13, 2011, at 3:58 PM, Ted Dunning wrote:
>>
>>> Essentially not.
>>>
>>> And I would worry about how to push the LSI vectors back into lucene in a
>>> coherent and usable way.
>>>
>>> On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org> wrote:
>>>
>>>> Is there some documentation/tutorial available on how to build a LSI
>>>> pipeline with mahout and lucene?
>>>>
>>>> --sebastian
>>>>
>>
>>
>>
>

Re: lsi

[Dmitriy Lyubimov <dl...@gmail.com>]

The only way to build model incrementally is to do a 'fold in' of new
observations, that i know.

However, folding in (which is just a multiplication of a new vector
over the matrices as Ted explained somewhere else) is just a
projection into already trained space of factors, but not a repetition
of a training, which is what the actual SVD is.

This may work in more ways than just for the sake of incremental
compilation. E.g. what i kind of doing: I am working on a particular
industry domain, so i look for a large corpus of documents pertinent
to that domain. That's what i run LSA on.

Then i actually throw away the entire document matrix and build a 100%
fold-in document index based on the term and singular value matrix.

The reason for this is that often the actual corpus of documents that
you are working with and need proximity comparisons to is not that big
at all to provide true picture of the specific domain you want to fit
the stuff into. So generic workflow that i figured in my case  is to
take a really big corpus relevant to your business, fit term
dictionary to it using LSA, and then use that dictionary to fold-in
new documents, with assumption that that big corpus is more relevant
and much bigger to what you want to focus on, than those documents you
actually want to retrieve and compare.

Then you may have one fairly infrequent training and a fairly simple
fold-in procedures.

On Thu, Nov 17, 2011 at 1:47 PM, Grant Ingersoll <gs...@apache.org> wrote:
> I've never implemented LSI.  Is there a way to incrementally build the model (by simply indexing documents) or is it something that one only runs after the fact once one has built up the much bigger matrix?  If it's the former, I bet it wouldn't be that hard to just implement the appropriate new codecs and similarity, assuming Lucene trunk.  If it's the latter, then Ted's comment about pushing back into Lucene gets a bit hairier.  Still, I wonder if the Codecs/Similarity could help here, too.
>
> What's a typical workflow look like for building all of this?
>
> On Nov 13, 2011, at 3:58 PM, Ted Dunning wrote:
>
>> Essentially not.
>>
>> And I would worry about how to push the LSI vectors back into lucene in a
>> coherent and usable way.
>>
>> On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org> wrote:
>>
>>> Is there some documentation/tutorial available on how to build a LSI
>>> pipeline with mahout and lucene?
>>>
>>> --sebastian
>>>
>
>
>

Re: lsi

[Grant Ingersoll <gs...@apache.org>]

I've never implemented LSI.  Is there a way to incrementally build the model (by simply indexing documents) or is it something that one only runs after the fact once one has built up the much bigger matrix?  If it's the former, I bet it wouldn't be that hard to just implement the appropriate new codecs and similarity, assuming Lucene trunk.  If it's the latter, then Ted's comment about pushing back into Lucene gets a bit hairier.  Still, I wonder if the Codecs/Similarity could help here, too.

What's a typical workflow look like for building all of this?

On Nov 13, 2011, at 3:58 PM, Ted Dunning wrote:

> Essentially not.
> 
> And I would worry about how to push the LSI vectors back into lucene in a
> coherent and usable way.
> 
> On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org> wrote:
> 
>> Is there some documentation/tutorial available on how to build a LSI
>> pipeline with mahout and lucene?
>> 
>> --sebastian
>> 

Re: lsi

[Jake Mannix <ja...@gmail.com>]

On Sun, Nov 13, 2011 at 10:31 PM, Ted Dunning <te...@gmail.com> wrote:
>
> I have done this with Lucene (some time ago) and had a hell of a time
> getting decent performance if I wanted to rescore a thousand documents from
> a disk based index.  That implies a memory based system again.  The cost of
> a thousand or so rescores is probably about a millisecond or so.  Since
> each vector is roughly a few cache lines in size, the achievable memory
> bandwidth should be significant.
>

Yeah, I guess I tend to make the assumption that everyone is all in memory
like I've been the past 4 years or so.  I have no idea what the current
Lucene
cost of looking up additional binary payloads from disk is while in the
inner
loop.  I could totally believe it's prohibitive.


>  Alternatively, to improve recall, at index-time, supplement each document
> > by terms in a new field "lsi_expanded" which are the terms closest in the
> > SVD projected space to the document, but aren't already in it.  Then at
> > query time, add an "... OR lsi_expanded:<query>" clause onto your query.
> > Instant query-expansion for recall enhancement.
> >
>
> This actually is pretty tricky to make well.
>

I never said it was necessarily a *good* idea to use LSI in this way (or,
in
fact, to use LSI at all), just that if you *do* have a good scoring model
(like
some kind of strongly predictive static prior, like PageRank), then doing
even fairly dumb recall-enhancing techniques can improve things quite
nicely, and "discretized" LSI like this is a "not completely dumb" way to
enhance recall.

  -jake

Re: lsi

[Ted Dunning <te...@gmail.com>]

On Sun, Nov 13, 2011 at 10:18 PM, Jake Mannix <ja...@gmail.com> wrote:

> On Sun, Nov 13, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > That handles coherent.
> >
> > IT doesn't handle usable.
> >
> > Storing the vectors as binary payloads handles the situation for
> > projection-like applications, but that doesn't help retrieval.
> >
>
> It's not just projection, it's for added relevance: if you are already
> doing
> Lucene for your scoring needs, you already are getting some good precision
> and recall.
>
> The idea is this: you take results you are *already* scoring, and add to
> that
> scoring function an LSI cosine as one feature among many.  Hopefully it
> will improve precision, even if it will do nothing for recall (as it's only
> being
> applied to results already retrieved by the text query).
>

I have done this with Lucene (some time ago) and had a hell of a time
getting decent performance if I wanted to rescore a thousand documents from
a disk based index.  That implies a memory based system again.  The cost of
a thousand or so rescores is probably about a millisecond or so.  Since
each vector is roughly a few cache lines in size, the achievable memory
bandwidth should be significant.


Alternatively, to improve recall, at index-time, supplement each document
> by terms in a new field "lsi_expanded" which are the terms closest in the
> SVD projected space to the document, but aren't already in it.  Then at
> query time, add an "... OR lsi_expanded:<query>" clause onto your query.
> Instant query-expansion for recall enhancement.
>

This actually is pretty tricky to make well.

Re: lsi

[Ken Krugler <kk...@transpac.com>]

We did something related for a recent project. Basically

 - Build a Lucene index using transformed data
 - Build the search query using similar transformations
 - Then take the top N, and do a more expensive scoring calculation

In the end, after much tweaking, it worked well - able to handle 1000 queries/sec on a biggish AWS box, by keeping everything in memory.

-- Ken

On Nov 13, 2011, at 10:18pm, Jake Mannix wrote:

> On Sun, Nov 13, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com> wrote:
> 
>> That handles coherent.
>> 
>> IT doesn't handle usable.
>> 
>> Storing the vectors as binary payloads handles the situation for
>> projection-like applications, but that doesn't help retrieval.
>> 
> 
> It's not just projection, it's for added relevance: if you are already doing
> Lucene for your scoring needs, you already are getting some good precision
> and recall.
> 
> The idea is this: you take results you are *already* scoring, and add to
> that
> scoring function an LSI cosine as one feature among many.  Hopefully it
> will improve precision, even if it will do nothing for recall (as it's only
> being
> applied to results already retrieved by the text query).
> 
> Alternatively, to improve recall, at index-time, supplement each document
> by terms in a new field "lsi_expanded" which are the terms closest in the
> SVD projected space to the document, but aren't already in it.  Then at
> query time, add an "... OR lsi_expanded:<query>" clause onto your query.
> Instant query-expansion for recall enhancement.
> 
> Or do both, and play with both your precision and recall.
> 
>  -jake

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

Re: lsi

[Lance Norskog <go...@gmail.com>]

A Lucene index uses effectively cosine distance, so it should be possible
to transform another cosine-based search into the Lucene format. This then
becomes new columns in an existing index. I have not worked out the origami
for this transformation.

Lance

On Sun, Nov 13, 2011 at 10:18 PM, Jake Mannix <ja...@gmail.com> wrote:

> On Sun, Nov 13, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > That handles coherent.
> >
> > IT doesn't handle usable.
> >
> > Storing the vectors as binary payloads handles the situation for
> > projection-like applications, but that doesn't help retrieval.
> >
>
> It's not just projection, it's for added relevance: if you are already
> doing
> Lucene for your scoring needs, you already are getting some good precision
> and recall.
>
> The idea is this: you take results you are *already* scoring, and add to
> that
> scoring function an LSI cosine as one feature among many.  Hopefully it
> will improve precision, even if it will do nothing for recall (as it's only
> being
> applied to results already retrieved by the text query).
>
> Alternatively, to improve recall, at index-time, supplement each document
> by terms in a new field "lsi_expanded" which are the terms closest in the
> SVD projected space to the document, but aren't already in it.  Then at
> query time, add an "... OR lsi_expanded:<query>" clause onto your query.
> Instant query-expansion for recall enhancement.
>
> Or do both, and play with both your precision and recall.
>
>  -jake
>



-- 
Lance Norskog
goksron@gmail.com

Re: lsi

[Jake Mannix <ja...@gmail.com>]

On Sun, Nov 13, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com> wrote:

> That handles coherent.
>
> IT doesn't handle usable.
>
> Storing the vectors as binary payloads handles the situation for
> projection-like applications, but that doesn't help retrieval.
>

It's not just projection, it's for added relevance: if you are already doing
Lucene for your scoring needs, you already are getting some good precision
and recall.

The idea is this: you take results you are *already* scoring, and add to
that
scoring function an LSI cosine as one feature among many.  Hopefully it
will improve precision, even if it will do nothing for recall (as it's only
being
applied to results already retrieved by the text query).

Alternatively, to improve recall, at index-time, supplement each document
by terms in a new field "lsi_expanded" which are the terms closest in the
SVD projected space to the document, but aren't already in it.  Then at
query time, add an "... OR lsi_expanded:<query>" clause onto your query.
Instant query-expansion for recall enhancement.

Or do both, and play with both your precision and recall.

  -jake

Re: lsi

[Ted Dunning <te...@gmail.com>]

It is really hard to answer that.  I would guess that it would be hard to
get much gain for uniformly distributed points in reasonably high
dimensional spaces (> 100 or so).  LSH, kd-trees and ball-trees all have
some difficulty at that size and the real kicker is that the logic and one
at a time operations required are massively slower than just bulk dot
products, especially if you can do multiple queries at the same time.

On the other hand, real data are almost never uniformly distributed so you
might wind up with very good speed gains with any of these techniques.
 Theory is difficult to apply in this case.

On Mon, Nov 14, 2011 at 6:48 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:

> Btw are there any known limitations on using in-memory kd trees to do knn
> scans?
>
> Their operational characteristics don't seem that bad.  Still probably not
> as good as some lsh based schemes but still should be quite good for many
> practical uses where N is millions and k is perhaps a thousand or so.
>
> For cosine and tanimoto stuff, it may need preliminary conversion to
> hypershperical.
>
> Also with the use of some improvements ( Max distance scan,
> bin-first-search) perhaps it may get to the point where it gets somewhat
> acceptable?
>  On Nov 13, 2011 10:10 PM, "Ted Dunning" <te...@gmail.com> wrote:
>
> > That handles coherent.
> >
> > IT doesn't handle usable.
> >
> > Storing the vectors as binary payloads handles the situation for
> > projection-like applications, but that doesn't help retrieval.
> >
> > The problem with retrieval is that you need to do a dot product for all
> > vectors.  THat used to require that you keep a low resolution version of
> > all of the vectors in memory and implement a very efficient dot product
> for
> > speed.  If you can batch up several queries (10 or more at a time), you
> can
> > get almost an order of magnitude improvement in retrieval speed since the
> > queries are almost always memory bound on the corpus vectors.
> >
> > With modern memory sizes, you can definitely hold some pretty big
> corpora.
> >  With 100 dimensions of floats, you can store two documents per kB which
> > gives 2 million documents per GB.  The problem really has more to do with
> > retrieval speed.  While the ALU can do multiplications and additions at a
> > pretty high rate, main memory can typically only be accessed at a rate
> of a
> > GB per second or so (or a few times that) which means that retrieval time
> > is going to be a significant fraction of a second or more for many
> corpora.
> >  That might be good enough, but high volume search engines typically need
> > search times of 10ms or better for the raw search engine which doesn't
> > allow for many vectors even if I am an order of magnitude off.
> >
> > The issue of memory bandwidth isn't getting better very quickly so there
> > isn't much hope for making this better.  A good inverted index running
> out
> > of memory can handle a LOT of documents and still meet the 10ms goal.
>  This
> > means that LSI as a raw search engine on large corpora is going to be
> less
> > cost effective by a couple of orders of magnitude.  That leaves lots of
> > room to use synthetic tokens, query expansion and other magic tricks in
> the
> > original inverted index search engine to achieve similar effects at much
> > less cost.
> >
> >
> > On Sun, Nov 13, 2011 at 3:56 PM, Jake Mannix <ja...@gmail.com>
> > wrote:
> >
> > > Store the vectors as binary payloads and keep the projection matrix in
> > > memory with the qureryBuilder, to add an "lsi cosine" between query and
> > doc
> > > scoring feature?
> > >
> > > On Nov 13, 2011 12:59 PM, "Ted Dunning" <te...@gmail.com> wrote:
> > >
> > > Essentially not.
> > >
> > > And I would worry about how to push the LSI vectors back into lucene
> in a
> > > coherent and usable way.
> > >
> > > On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org>
> > > wrote: > Is there some docum...
> > >
> >
>

Re: lsi

[Dmitriy Lyubimov <dl...@gmail.com>]

Btw are there any known limitations on using in-memory kd trees to do knn
scans?

Their operational characteristics don't seem that bad.  Still probably not
as good as some lsh based schemes but still should be quite good for many
practical uses where N is millions and k is perhaps a thousand or so.

For cosine and tanimoto stuff, it may need preliminary conversion to
hypershperical.

Also with the use of some improvements ( Max distance scan,
bin-first-search) perhaps it may get to the point where it gets somewhat
acceptable?
 On Nov 13, 2011 10:10 PM, "Ted Dunning" <te...@gmail.com> wrote:

> That handles coherent.
>
> IT doesn't handle usable.
>
> Storing the vectors as binary payloads handles the situation for
> projection-like applications, but that doesn't help retrieval.
>
> The problem with retrieval is that you need to do a dot product for all
> vectors.  THat used to require that you keep a low resolution version of
> all of the vectors in memory and implement a very efficient dot product for
> speed.  If you can batch up several queries (10 or more at a time), you can
> get almost an order of magnitude improvement in retrieval speed since the
> queries are almost always memory bound on the corpus vectors.
>
> With modern memory sizes, you can definitely hold some pretty big corpora.
>  With 100 dimensions of floats, you can store two documents per kB which
> gives 2 million documents per GB.  The problem really has more to do with
> retrieval speed.  While the ALU can do multiplications and additions at a
> pretty high rate, main memory can typically only be accessed at a rate of a
> GB per second or so (or a few times that) which means that retrieval time
> is going to be a significant fraction of a second or more for many corpora.
>  That might be good enough, but high volume search engines typically need
> search times of 10ms or better for the raw search engine which doesn't
> allow for many vectors even if I am an order of magnitude off.
>
> The issue of memory bandwidth isn't getting better very quickly so there
> isn't much hope for making this better.  A good inverted index running out
> of memory can handle a LOT of documents and still meet the 10ms goal.  This
> means that LSI as a raw search engine on large corpora is going to be less
> cost effective by a couple of orders of magnitude.  That leaves lots of
> room to use synthetic tokens, query expansion and other magic tricks in the
> original inverted index search engine to achieve similar effects at much
> less cost.
>
>
> On Sun, Nov 13, 2011 at 3:56 PM, Jake Mannix <ja...@gmail.com>
> wrote:
>
> > Store the vectors as binary payloads and keep the projection matrix in
> > memory with the qureryBuilder, to add an "lsi cosine" between query and
> doc
> > scoring feature?
> >
> > On Nov 13, 2011 12:59 PM, "Ted Dunning" <te...@gmail.com> wrote:
> >
> > Essentially not.
> >
> > And I would worry about how to push the LSI vectors back into lucene in a
> > coherent and usable way.
> >
> > On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org>
> > wrote: > Is there some docum...
> >
>

Re: lsi

[Ted Dunning <te...@gmail.com>]

That handles coherent.

IT doesn't handle usable.

Storing the vectors as binary payloads handles the situation for
projection-like applications, but that doesn't help retrieval.

The problem with retrieval is that you need to do a dot product for all
vectors.  THat used to require that you keep a low resolution version of
all of the vectors in memory and implement a very efficient dot product for
speed.  If you can batch up several queries (10 or more at a time), you can
get almost an order of magnitude improvement in retrieval speed since the
queries are almost always memory bound on the corpus vectors.

With modern memory sizes, you can definitely hold some pretty big corpora.
 With 100 dimensions of floats, you can store two documents per kB which
gives 2 million documents per GB.  The problem really has more to do with
retrieval speed.  While the ALU can do multiplications and additions at a
pretty high rate, main memory can typically only be accessed at a rate of a
GB per second or so (or a few times that) which means that retrieval time
is going to be a significant fraction of a second or more for many corpora.
 That might be good enough, but high volume search engines typically need
search times of 10ms or better for the raw search engine which doesn't
allow for many vectors even if I am an order of magnitude off.

The issue of memory bandwidth isn't getting better very quickly so there
isn't much hope for making this better.  A good inverted index running out
of memory can handle a LOT of documents and still meet the 10ms goal.  This
means that LSI as a raw search engine on large corpora is going to be less
cost effective by a couple of orders of magnitude.  That leaves lots of
room to use synthetic tokens, query expansion and other magic tricks in the
original inverted index search engine to achieve similar effects at much
less cost.


On Sun, Nov 13, 2011 at 3:56 PM, Jake Mannix <ja...@gmail.com> wrote:

> Store the vectors as binary payloads and keep the projection matrix in
> memory with the qureryBuilder, to add an "lsi cosine" between query and doc
> scoring feature?
>
> On Nov 13, 2011 12:59 PM, "Ted Dunning" <te...@gmail.com> wrote:
>
> Essentially not.
>
> And I would worry about how to push the LSI vectors back into lucene in a
> coherent and usable way.
>
> On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org>
> wrote: > Is there some docum...
>

Re: lsi

[Jake Mannix <ja...@gmail.com>]

Store the vectors as binary payloads and keep the projection matrix in
memory with the qureryBuilder, to add an "lsi cosine" between query and doc
scoring feature?

On Nov 13, 2011 12:59 PM, "Ted Dunning" <te...@gmail.com> wrote:

Essentially not.

And I would worry about how to push the LSI vectors back into lucene in a
coherent and usable way.

On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org>
wrote: > Is there some docum...

Re: lsi

[Ted Dunning <te...@gmail.com>]

Essentially not.

And I would worry about how to push the LSI vectors back into lucene in a
coherent and usable way.

On Sun, Nov 13, 2011 at 10:47 AM, Sebastian Schelter <ss...@apache.org> wrote:

> Is there some documentation/tutorial available on how to build a LSI
> pipeline with mahout and lucene?
>
> --sebastian
>

Re: lsi

[Dmitriy Lyubimov <dl...@gmail.com>]

I use seq2sparse + ssvd. Subsequent use patterns vary and in my case
proprietary but mainly revolve around fold-in updates into pretrained term
space and various locality sensitive tricks depending on the patterns you
use. My pattern involves scanning  first n nearest neighbours with smallest
distance first preferably without examining the entire neighborhood as
opposed to finding all neighbours in a given distance radius which is what
most of algorithms actually do out of the box.

I suspect although not quite convinced that document mixture models such as
lda would produce a better fit than classic svd based lsi.
On Nov 13, 2011 10:48 AM, "Sebastian Schelter" <ss...@apache.org> wrote:

> Is there some documentation/tutorial available on how to build a LSI
> pipeline with mahout and lucene?
>
> --sebastian
>

Re: lsi

[Grant Ingersoll <gs...@apache.org>]

Might be useful: https://github.com/algoriffic/lsa4solr

Looks like it hasn't been kept up to date.

On Nov 13, 2011, at 1:47 PM, Sebastian Schelter wrote:

> Is there some documentation/tutorial available on how to build a LSI
> pipeline with mahout and lucene?
> 
> --sebastian