You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by webdev1977 <we...@gmail.com> on 2010/12/06 17:44:07 UTC
Taxonomy and Faceting
I have been digging through the user lists for Solr and Nutch, as well as
reading lots of blogs, etc. I have yet to find a clear answer (maybe there
is none )
I am trying to find the best way ahead for choosing a technology that will
allow the ability to use a large taxonomy for classifying structured and
unstructured data and then displaying those categorizations as facets to the
user during search.
There seems to be several approaches, some of which make use of index time
for encoding the terms found in the text, but I have seen no mention of HOW
to get those terms from the text. Some sort of text classification software
I am assuming. If this is true, are there any good open source engines that
can process text against a taxonomy?
The other approach seems to be two patches being developed for Solr 3.0, 792
and 64. Again, I think you would have to have some sort of an engine to
give you this information that could then be added at index time.
I have also seen some interesting literature on using Drupal and the Solr
module.
My current architecture uses Nutch (1.2) for crawling, solrindex for inexing
(Solr 1.4.1), and Ajax Solr for my UI.
I have also seen some talk in webinars/etc from Lucid Imagination about
upcoming development on "Native Taxonomy Facets", any idea where that
development stands?
I have to use the most stable version of Solr/Nutch/Lucene possible for my
implementation, because, unfortunately, once I choose, going back will be
next to impossible for years to come!
Thanks!
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2028442.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by Tommaso Teofili <to...@gmail.com>.
With the SOLR-2129 patch you enable an Apache UIMA [1] pipeline to enrich
documents being indexed.
The base pipeline provided with the patch uses the following blocks (see
OverridingParamsExtServicesAE.xml):
<node>AggregateSentenceAE</node>
<node>OpenCalaisAnnotator</node>
<node>TextKeywordExtractionAEDescriptor</node>
<node>TextLanguageDetectionAEDescriptor</node>
<node>TextCategorizationAEDescriptor</node>
<node>TextConceptTaggingAEDescriptor</node>
<node>TextRankedEntityExtractionAEDescriptor</node>
This enables tokenizing, adding part of speech to tokens extract sentences
with WhitespaceTokenizer and HMMTagger, then inserts named entities and
language extracted with OpenCalaisAnnotator and AlchemyAPIAnnotator.
The parameters you underlined are relevant only if you use
OpenCalaisAnnotator and AlchemyAPIAnnotator; as you may see those are
runtime parameters, so depending on which Analysis Engine you're executing
you could need or not such parameters or need other ones.
However you can change the pipeline blocks to use to whatever you want,
provided that they are UIMA compliant specifying the relative Analysis
Engine descriptor inside the tag:
<analysisEngine>/org/apache/uima/desc
/OverridingParamsExtServicesAE.xml</analysisEngine>.
There are many other engines you can use and configure with SOLR-2129, see
[2] and [3].
I hope this clarifies things a little more.
Cheers,
Tommaso
[1] : http://uima.apache.org
[2] : http://uima.apache.org/sandbox.html
[3] : http://uima.apache.org/external-resources.html
2010/12/13 webdev1977 <we...@gmail.com>
>
> Based on this:
>
> <keyword_apikey>VALID_ALCHEMYAPI_KEY</keyword_apikey>
>
> <concept_apikey>VALID_ALCHEMYAPI_KEY</concept_apikey>
>
> <lang_apikey>VALID_ALCHEMYAPI_KEY</lang_apikey>
>
> <cat_apikey>VALID_ALCHEMYAPI_KEY</cat_apikey>
>
> <entities_apikey>VALID_ALCHEMYAPI_KEY</entities_apikey>
>
> <oc_licenseID>VALID_OPENCALAIS_KEY</oc_licenseID>
>
>
> ...this can't be used unless you use some sort of processing engine? I am
> playing around with some other open source tagging software, but I have yet
> to get very far.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2079148.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Taxonomy and Faceting
Posted by webdev1977 <we...@gmail.com>.
Based on this:
<keyword_apikey>VALID_ALCHEMYAPI_KEY</keyword_apikey>
<concept_apikey>VALID_ALCHEMYAPI_KEY</concept_apikey>
<lang_apikey>VALID_ALCHEMYAPI_KEY</lang_apikey>
<cat_apikey>VALID_ALCHEMYAPI_KEY</cat_apikey>
<entities_apikey>VALID_ALCHEMYAPI_KEY</entities_apikey>
<oc_licenseID>VALID_OPENCALAIS_KEY</oc_licenseID>
...this can't be used unless you use some sort of processing engine? I am
playing around with some other open source tagging software, but I have yet
to get very far.
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2079148.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by Tommaso Teofili <to...@gmail.com>.
Thanks Markus for helping with that, there are some changes in the
configuration that need to be done.
However I've just submitted a new patch at [1] which fix jar packaging and
holds a README.txt which contains the following, it's very simple :
1. copy generated solr-uima jar and its libs (under contrib/uima/lib)
inside a Solr libraries directory.
2. modify your schema.xml adding the fields you want to be hold
metadataspecifying proper values for type, indexed, stored and
multiValued options:
3. for example you could specify the following
<field name="language" type="string" indexed="true" stored="true"
required="false"/>
<field name="concept" type="string" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="sentence" type="text" indexed="true" stored="true"
multiValued="true" required="false" />
4. modify your solrconfig.xml adding the following snippet:
<uimaConfig>
<runtimeParameters>
<keyword_apikey>VALID_ALCHEMYAPI_KEY</keyword_apikey>
<concept_apikey>VALID_ALCHEMYAPI_KEY</concept_apikey>
<lang_apikey>VALID_ALCHEMYAPI_KEY</lang_apikey>
<cat_apikey>VALID_ALCHEMYAPI_KEY</cat_apikey>
<entities_apikey>VALID_ALCHEMYAPI_KEY</entities_apikey>
<oc_licenseID>VALID_OPENCALAIS_KEY</oc_licenseID>
</runtimeParameters>
<analysisEngine>/org/apache/uima/desc
/OverridingParamsExtServicesAE.xml</analysisEngine>
<analyzeFields merge="false">text</analyzeFields>
<fieldMapping>
<type name="org.apache.uima.alchemy.ts.concept.ConceptFS">
<map feature="text" field="concept"/>
</type>
<type name="org.apache.uima.alchemy.ts.language.LanguageFS">
<map feature="language" field="language"/>
</type>
<type name="org.apache.uima.SentenceAnnotation">
<map feature="coveredText" field="sentence"/>
</type>
</fieldMapping>
</uimaConfig>
5. the analysisEngine tag must contain an AE descriptor inside the specified
path in the classpath
6. the analyzeFields tag must contain the input fields that need to be
analyzed by UIMA,
if merge=true then their content will be merged and analyzed only once
7. field mapping describes which features of which types should go in a
field
8. define in your solrconfig.xml an UpdateRequestProcessorChain as
following:
<updateRequestProcessorChain name="uima">
<processor class="org.apache.solr.uima.processor.UIMAProcessorFactory"/>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
9. in your solrconfig.xml replace the existing default (<requestHandler
name="/update"...) or create a new UpdateRequestHandler with the following:
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
<lst name="defaults">
<str name="update.processor">uima</str>
</lst>
</requestHandler>
Hope this helps,
Tommaso
[1] : https://issues.apache.org/jira/browse/SOLR-2129
2010/12/8 Markus Jelsma <ma...@openindex.io>
> Don't know if its useful but from the old thread:
> http://code.google.com/p/solr-uima/wiki/5MinutesTutorial
>
>
> On Wednesday 08 December 2010 16:18:06 webdev1977 wrote:
> > Any luck with a tutorial? :-)
>
> --
> Markus Jelsma - CTO - Openindex
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
>
Re: Taxonomy and Faceting
Posted by Markus Jelsma <ma...@openindex.io>.
Don't know if its useful but from the old thread:
http://code.google.com/p/solr-uima/wiki/5MinutesTutorial
On Wednesday 08 December 2010 16:18:06 webdev1977 wrote:
> Any luck with a tutorial? :-)
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
Re: Taxonomy and Faceting
Posted by webdev1977 <we...@gmail.com>.
Any luck with a tutorial? :-)
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2040246.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by webdev1977 <we...@gmail.com>.
That would be AMAZING!! And much appreciated ;-)
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2033657.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by Tommaso Teofili <to...@gmail.com>.
Hi,
as I made the patch I can guide you through the Solr-UIMA integration
configuration, just give me some more time as I am really busy at the moment
and can't deepen it. There was a mini tutorial but it's outdated, I'll
update it and let you know here in a few hours.
Cheers,
Tommaso
2010/12/7 webdev1977 <we...@gmail.com>
>
> Can someone enlighten me on how to get started with this patch? I am
> running
> solr 1.4.1 and I need to download the latest trunk and apply the patch
> obviously.. But after that, I am sort of clueless.. I am assuming there are
> some things that have to happen in solr config and schema files.
>
> Reading the code now...
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2033563.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Taxonomy and Faceting
Posted by webdev1977 <we...@gmail.com>.
Can someone enlighten me on how to get started with this patch? I am running
solr 1.4.1 and I need to download the latest trunk and apply the patch
obviously.. But after that, I am sort of clueless.. I am assuming there are
some things that have to happen in solr config and schema files.
Reading the code now...
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2033563.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by Lance Norskog <go...@gmail.com>.
That is correct. Solr is a search engine, not a text analysis engine.
There are a few open source text analysis systems: Weka, OpenNLP,
UIMA.
Someone is working on integrating UIMA with Solr:
https://issues.apache.org/jira/browse/SOLR-2129
But you should generally assume you will have a batch processing pass
over the data before indexing it.
On Mon, Dec 6, 2010 at 12:04 PM, webdev1977 <we...@gmail.com> wrote:
>
> Thanks for the quick response!
>
> I was thinking more about the idea of having both structured and unstructred
> data coming into a system to be indexed/searched. I would like these
> documents to be processed by some sort of entity/keyword/semantic
> processing. I have a well defined taxonomy for my organization (it is quite
> large) and at the moment we use RetrievalWare to give keyword/classification
> suggestions. This does NOT work well though, and RetrievalWare is pretty
> much useless to us.
>
> I want a way to do this process either at index time or search time. All
> documents should be processed against this taxonomy. I do not want the user
> to be able to nominate keywords, it must happen automatically. I am
> assuming it is only natural for these keywords/taxonomy entities to show up
> as hierarchical facets?
>
> From what I can tell, there is no way to tell Solr.. here is my taxonomy..
> classify my documents and give me back facets and facet counts..
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2029636.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Lance Norskog
goksron@gmail.com
Re: Taxonomy and Faceting
Posted by webdev1977 <we...@gmail.com>.
Thanks for the quick response!
I was thinking more about the idea of having both structured and unstructred
data coming into a system to be indexed/searched. I would like these
documents to be processed by some sort of entity/keyword/semantic
processing. I have a well defined taxonomy for my organization (it is quite
large) and at the moment we use RetrievalWare to give keyword/classification
suggestions. This does NOT work well though, and RetrievalWare is pretty
much useless to us.
I want a way to do this process either at index time or search time. All
documents should be processed against this taxonomy. I do not want the user
to be able to nominate keywords, it must happen automatically. I am
assuming it is only natural for these keywords/taxonomy entities to show up
as hierarchical facets?
>From what I can tell, there is no way to tell Solr.. here is my taxonomy..
classify my documents and give me back facets and facet counts..
--
View this message in context: http://lucene.472066.n3.nabble.com/Taxonomy-and-Faceting-tp2028442p2029636.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Taxonomy and Faceting
Posted by Peter Karich <pe...@yahoo.de>.
I'm unsure but maybe you mean something like clustering? Then carrot^2
can do this (at index time I think):
http://search.carrot2.org/stable/search?query=jetwick&view=visu
(There is a plugin for solr)
Or do you already know the categories of your docs. E.g. you already
have a category tree and associated documents?
Regards,
Peter.
> I have been digging through the user lists for Solr and Nutch, as well as
> reading lots of blogs, etc. I have yet to find a clear answer (maybe there
> is none )
>
> I am trying to find the best way ahead for choosing a technology that will
> allow the ability to use a large taxonomy for classifying structured and
> unstructured data and then displaying those categorizations as facets to the
> user during search.
>
> There seems to be several approaches, some of which make use of index time
> for encoding the terms found in the text, but I have seen no mention of HOW
> to get those terms from the text. Some sort of text classification software
> I am assuming. If this is true, are there any good open source engines that
> can process text against a taxonomy?
>
> The other approach seems to be two patches being developed for Solr 3.0, 792
> and 64. Again, I think you would have to have some sort of an engine to
> give you this information that could then be added at index time.
>
> I have also seen some interesting literature on using Drupal and the Solr
> module.
>
> My current architecture uses Nutch (1.2) for crawling, solrindex for inexing
> (Solr 1.4.1), and Ajax Solr for my UI.
>
> I have also seen some talk in webinars/etc from Lucid Imagination about
> upcoming development on "Native Taxonomy Facets", any idea where that
> development stands?
>
> I have to use the most stable version of Solr/Nutch/Lucene possible for my
> implementation, because, unfortunately, once I choose, going back will be
> next to impossible for years to come!
>
> Thanks!