You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Neal Richter <nr...@gmail.com> on 2009/01/26 18:29:36 UTC
Text classification with Solr
Hey all,
I'm in the processing of implementing a system to do 'text
classification' with Solr. The basic idea is to take an
ontology/taxonomy like dmoz of {label: "X", tags: "a,b,c,d,e"}, index
it and then classify documents into the taxonomy by pushing parsed
document into the Solr search API. Why? Lucene/Solr's ability to do
weighted term boosting at both search and index time has lots of
obvious uses here.
Has anyone worked on this or a similar project yet? I've seen some
talk on the list about this area but it's pretty thin... December
thread "Taxonomy Support on Solr". I'm assuming Grant Ingersoll is
looking at similar things with his 'taming text' project.
I store the 'documents' in another repository and they are far too
dynamic (write intensive) for direct indexing in Solr... so the
previously suggested procedure of 1) store document 2) execute
more-like-this and 3) delete document would be too slow.
If people are interested I could start a JIRA issue on this (I do not
see anything there at the moment).
Thanks - Neal Richter
http://aicoder.blogspot.com
Re: Text classification with Solr
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Jan 26, 2009 at 10:59 PM, Neal Richter <nr...@gmail.com> wrote:
> Hey all,
>
> I'm in the processing of implementing a system to do 'text
> classification' with Solr. The basic idea is to take an
> ontology/taxonomy like dmoz of {label: "X", tags: "a,b,c,d,e"}, index
> it and then classify documents into the taxonomy by pushing parsed
> document into the Solr search API. Why? Lucene/Solr's ability to do
> weighted term boosting at both search and index time has lots of
> obvious uses here.
>
> Has anyone worked on this or a similar project yet? I've seen some
> talk on the list about this area but it's pretty thin... December
> thread "Taxonomy Support on Solr". I'm assuming Grant Ingersoll is
> looking at similar things with his 'taming text' project.
>
> I store the 'documents' in another repository and they are far too
> dynamic (write intensive) for direct indexing in Solr... so the
> previously suggested procedure of 1) store document 2) execute
> more-like-this and 3) delete document would be too slow.
>
> If people are interested I could start a JIRA issue on this (I do not
> see anything there at the moment).
>
> Thanks - Neal Richter
> http://aicoder.blogspot.com
>
Grant did some work at https://issues.apache.org/jira/browse/SOLR-769
Take a look and see if that helps.
--
Regards,
Shalin Shekhar Mangar.
Re: Text classification with Solr
Posted by Hannes Carl Meyer <ma...@hcmeyer.com>.
>From my past projects, our Lucene classification corpus looked like this:
0|document text...|categoryA
1|document text...|categoryB
2|document text...|categoryA
3|document text...|categoryA
...
800|document text...|categoryC
With the faceting capabilities of Solr it is now possible to design more
dimensions of categories/taxonomies in a corpus with a minimal impact (?) on
computation time! Plus the configuration of synonyms in Solr configuration.
Like the idea to use Solr!
On Wed, Jan 28, 2009 at 7:57 AM, Neal Richter <nr...@gmail.com> wrote:
> On Tue, Jan 27, 2009 at 2:21 PM, Grant Ingersoll <gs...@apache.org>
> wrote:
> > One of the things I am interested in is the marriage of Solr and Mahout
> > (which has some Genetic Algorithms support) and other ML (Weka, etc.)
> tools.
> [snip]
>
> I love it, good to know you are thinking big here. Here's another big
> thought:
> http://www.eml-r.org/nlp/papers/ponzetto07b.pdf .. but assume we want
> to extract this type of structure from the full text of Wikipedia
> rather than the narrow categories DB.
>
> > Things that can help with all this: LukeReqHandler, TermVectorComponent,
> > TermsComponent, others
> >
>
> [snip]
>
> > Neal, what did you have in mind for a JIRA issue? I'd love to see a
> patch.
>
> More research needed, but the initial idea would be to enable the
> passing in of a weighted term vector as a query and allowing a
> more-like-this type search on it. Anyone attempt this yet?
>
> Interesting point about faceting here is that it would give outgoing
> feedback on what /new/ words (not in initial query) that if added to
> the query would result in additional discrimination between the
> matched categories.
>
> So Solr outputs a set of categories for a document, and also emits a
> set of related words to the initial query! Categorization and
> recommendation in one.
>
> - Neal
>
Re: Text classification with Solr
Posted by Neal Richter <nr...@gmail.com>.
On Tue, Jan 27, 2009 at 2:21 PM, Grant Ingersoll <gs...@apache.org> wrote:
> One of the things I am interested in is the marriage of Solr and Mahout
> (which has some Genetic Algorithms support) and other ML (Weka, etc.) tools.
[snip]
I love it, good to know you are thinking big here. Here's another big thought:
http://www.eml-r.org/nlp/papers/ponzetto07b.pdf .. but assume we want
to extract this type of structure from the full text of Wikipedia
rather than the narrow categories DB.
> Things that can help with all this: LukeReqHandler, TermVectorComponent,
> TermsComponent, others
>
[snip]
> Neal, what did you have in mind for a JIRA issue? I'd love to see a patch.
More research needed, but the initial idea would be to enable the
passing in of a weighted term vector as a query and allowing a
more-like-this type search on it. Anyone attempt this yet?
Interesting point about faceting here is that it would give outgoing
feedback on what /new/ words (not in initial query) that if added to
the query would result in additional discrimination between the
matched categories.
So Solr outputs a set of categories for a document, and also emits a
set of related words to the initial query! Categorization and
recommendation in one.
- Neal
Re: Text classification with Solr
Posted by Grant Ingersoll <gs...@apache.org>.
I guess I've been called to the chalkboard...
I haven't looked specifically at putting the taxonomy in Lucene/Solr,
but it is an interesting idea. In reading the paper you mentioned,
there are some interesting ideas there and Solr could obviously just
as easily be used as Lucene, I think.
One of the things I am interested in is the marriage of Solr and
Mahout (which has some Genetic Algorithms support) and other ML (Weka,
etc.) tools. So, for instance in the paper, they have multiple
indexes, one for negative and positive sets, well that could be done
with Solr cores or just through intelligent filtering. Then, you
could have Mahout work do it's training/clustering/whatever in the
background as needed just by sending a ReqHandler commands and output
it's model that can be shared on the "output" side so that you can
nicely serve up your results as part of search results or even
standalone, so either as a SearchComponent or from the ReqHandler. Of
course, the tricky part is in the implementation and managing the
memory, threading, etc.
Things that can help with all this: LukeReqHandler,
TermVectorComponent, TermsComponent, others
As for Hannes question about "Why Solr" I think you can still get
close to the metal w/ Solr just as Lucene, but now you have the built
in framework that makes experimentation so much easier, IMO, plus you
have all the features that Solr has to offer. For instance, a
reasonable thing to do with the output from the classification is, of
course, to facet on them.
Neal, what did you have in mind for a JIRA issue? I'd love to see a
patch.
On Jan 26, 2009, at 12:29 PM, Neal Richter wrote:
> Hey all,
>
> I'm in the processing of implementing a system to do 'text
> classification' with Solr. The basic idea is to take an
> ontology/taxonomy like dmoz of {label: "X", tags: "a,b,c,d,e"}, index
> it and then classify documents into the taxonomy by pushing parsed
> document into the Solr search API. Why? Lucene/Solr's ability to do
> weighted term boosting at both search and index time has lots of
> obvious uses here.
>
> Has anyone worked on this or a similar project yet? I've seen some
> talk on the list about this area but it's pretty thin... December
> thread "Taxonomy Support on Solr". I'm assuming Grant Ingersoll is
> looking at similar things with his 'taming text' project.
>
> I store the 'documents' in another repository and they are far too
> dynamic (write intensive) for direct indexing in Solr... so the
> previously suggested procedure of 1) store document 2) execute
> more-like-this and 3) delete document would be too slow.
>
> If people are interested I could start a JIRA issue on this (I do not
> see anything there at the moment).
>
> Thanks - Neal Richter
> http://aicoder.blogspot.com
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Re: Text classification with Solr
Posted by Karl Wettin <ka...@gmail.com>.
27 jan 2009 kl. 17.23 skrev Neal Richter:
>>
>> Is it really neccessary to use Solr for it? Things going much
>> faster with
>> Lucene low-level api and much faster if you're loading the
>> classification
>> corpus into the RAM.
>
> Good points. At the moment I'd rather have a daemon with a service
> API.. as well as the filtering/tokenization capabilities Solr has
> built in. Probably will attempt to get the corpus' index in memory
> via large memory allocation.
>
> If it doesn't scale then I'll either go to Lucene api or implement a
> custom inverted index via memcached.
>
> Other note /at the moment/ is that it's not going to be a deeply
> hierarchical taxonomy, much less a full indexing of an RDF/OWL
> schema.. there are some gotchas for that.
If your corpus is small enought you may want to take a look at lucene/
contrib/instantiated. It was made just for these sort of things.
karl
Re: Text classification with Solr
Posted by Neal Richter <nr...@gmail.com>.
On Tue, Jan 27, 2009 at 1:36 AM, Hannes Carl Meyer <ma...@hcmeyer.com> wrote:
> Yeah, know it, the challenge on this method is the calculation of the score
> and parametrization of thresholds.
Not as worried about score itself as the score thresholds for prediction in/out.
> Is it really neccessary to use Solr for it? Things going much faster with
> Lucene low-level api and much faster if you're loading the classification
> corpus into the RAM.
Good points. At the moment I'd rather have a daemon with a service
API.. as well as the filtering/tokenization capabilities Solr has
built in. Probably will attempt to get the corpus' index in memory
via large memory allocation.
If it doesn't scale then I'll either go to Lucene api or implement a
custom inverted index via memcached.
Other note /at the moment/ is that it's not going to be a deeply
hierarchical taxonomy, much less a full indexing of an RDF/OWL
schema.. there are some gotchas for that.
Thanks - Neal
Re: Text classification with Solr
Posted by Hannes Carl Meyer <ma...@hcmeyer.com>.
>>Instead of indexing documents about 'sports' and searching for hits
>>based upon 'basketball', 'football' etc.. I simply want to index the
>>taxonomy and classify documents into it. This is a an ancient
>>AI/Data-Mining discipline.. but the standard methods of 'indexing' the
>>taxonomy are/were primitive compared to what one /could/ do with
>>something like Lucene.
Yeah, know it, the challenge on this method is the calculation of the score
and parametrization of thresholds.
Is it really neccessary to use Solr for it? Things going much faster with
Lucene low-level api and much faster if you're loading the classification
corpus into the RAM.
On Mon, Jan 26, 2009 at 7:24 PM, Neal Richter <nr...@gmail.com> wrote:
> Thanks for the link Shalin... played with that a while back.. It's
> possibly got some indirect possibilities.
>
> On Mon, Jan 26, 2009 at 10:46 AM, Hannes Carl Meyer <ma...@hcmeyer.com>
> wrote:
> > I didn't understand, is the corpus of documents you want to use to
> classify
> > fix?
>
> Assume the 'documents' are not stored in the same index and I want to
> only store the taxonomy or ontology in this index.
>
> Instead of indexing documents about 'sports' and searching for hits
> based upon 'basketball', 'football' etc.. I simply want to index the
> taxonomy and classify documents into it. This is a an ancient
> AI/Data-Mining discipline.. but the standard methods of 'indexing' the
> taxonomy are/were primitive compared to what one /could/ do with
> something like Lucene.
>
> Here's a 2007 research paper that used Lucene directly for
> classification, but doing the inverse of what I described:
> http://www.cs.ucl.ac.uk/staff/R.Hirsch/papers/gecco_HHS.pdf
>
> >>>previously suggested procedure of 1) store document 2) execute
> >>>more-like-this and 3) delete document would be too slow.
> > Do you mean the document to classify?
> > Why do you then want to put it into the index (very expensive), you just
> > need the contents of it to build a query!
>
> Exactly.. in the December Taxonomy thread Walter Underwood outlined a
> store/classify/delete procedure. Too slow if you have no need to
> index the document itself.
>
> - Neal
>
Re: Text classification with Solr
Posted by Neal Richter <nr...@gmail.com>.
Thanks for the link Shalin... played with that a while back.. It's
possibly got some indirect possibilities.
On Mon, Jan 26, 2009 at 10:46 AM, Hannes Carl Meyer <ma...@hcmeyer.com> wrote:
> I didn't understand, is the corpus of documents you want to use to classify
> fix?
Assume the 'documents' are not stored in the same index and I want to
only store the taxonomy or ontology in this index.
Instead of indexing documents about 'sports' and searching for hits
based upon 'basketball', 'football' etc.. I simply want to index the
taxonomy and classify documents into it. This is a an ancient
AI/Data-Mining discipline.. but the standard methods of 'indexing' the
taxonomy are/were primitive compared to what one /could/ do with
something like Lucene.
Here's a 2007 research paper that used Lucene directly for
classification, but doing the inverse of what I described:
http://www.cs.ucl.ac.uk/staff/R.Hirsch/papers/gecco_HHS.pdf
>>>previously suggested procedure of 1) store document 2) execute
>>>more-like-this and 3) delete document would be too slow.
> Do you mean the document to classify?
> Why do you then want to put it into the index (very expensive), you just
> need the contents of it to build a query!
Exactly.. in the December Taxonomy thread Walter Underwood outlined a
store/classify/delete procedure. Too slow if you have no need to
index the document itself.
- Neal
Re: Text classification with Solr
Posted by Hannes Carl Meyer <ma...@hcmeyer.com>.
Hi Neal,
this sounds pretty similar to me. Did a lot of those projects some years ago
(with Lucene low-level API)!
I didn't understand, is the corpus of documents you want to use to classify
fix?
>>previously suggested procedure of 1) store document 2) execute
>>more-like-this and 3) delete document would be too slow.
Do you mean the document to classify?
Why do you then want to put it into the index (very expensive), you just
need the contents of it to build a query!
Regards
Hannes
On Mon, Jan 26, 2009 at 6:29 PM, Neal Richter <nr...@gmail.com> wrote:
> Hey all,
>
> I'm in the processing of implementing a system to do 'text
> classification' with Solr. The basic idea is to take an
> ontology/taxonomy like dmoz of {label: "X", tags: "a,b,c,d,e"}, index
> it and then classify documents into the taxonomy by pushing parsed
> document into the Solr search API. Why? Lucene/Solr's ability to do
> weighted term boosting at both search and index time has lots of
> obvious uses here.
>
> Has anyone worked on this or a similar project yet? I've seen some
> talk on the list about this area but it's pretty thin... December
> thread "Taxonomy Support on Solr". I'm assuming Grant Ingersoll is
> looking at similar things with his 'taming text' project.
>
> I store the 'documents' in another repository and they are far too
> dynamic (write intensive) for direct indexing in Solr... so the
> previously suggested procedure of 1) store document 2) execute
> more-like-this and 3) delete document would be too slow.
>
> If people are interested I could start a JIRA issue on this (I do not
> see anything there at the moment).
>
> Thanks - Neal Richter
> http://aicoder.blogspot.com
>