You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Wunderlich, Tobias" <to...@igd-r.fraunhofer.de> on 2012/05/24 14:04:12 UTC
Creating custom Filter / Tokenizer / Request Handler for
integration of NER-Framework
Hey Guys,
I am recently working on a project to integrate a Named-Entity-Recognition-Framework (NER) in an existing searchplatform based on Solr. The Platform uses ManifoldCF to automatically gather the content from various repositories. The NER-Framework creates Annotations/Metadata from given content which I then want to integrate into the search-platform as metadata to use for faceting. Since MCF handles all content gathering, I need a way to integrate the NER-Framework directly into Solr. The Goal is to get all Annotations per document into a multivalued field. My first thought was to create a custom filter, which just takes the content and gives back only the Annotations. But as I understand it, a filter only processes predetermined Tokens, which is useless for my purpose, since the NER-Framework needs to process the whole content of a document. What about a custom Tokenizer? Would it be possible to process the whole text and give back only the Annotations as Tokens? A third thought was to manipulate the ExtractRequestHandler (Solr Cell) used by MCF to somehow add the Annotations as Metadata when the content and metadata is distributed to the different fields.
I hope my problem description is sufficient. Does anybody have any thoughts on that subject?
Best regards,
Tobias
AW: Creating custom Filter / Tokenizer / Request Handler for
integration of NER-Framework
Posted by "Wunderlich, Tobias" <to...@igd-r.fraunhofer.de>.
Thanks for all the responses. I went with the UpdateRequestProcessor and it works.
-----Ursprüngliche Nachricht-----
Von: Lance Norskog [mailto:goksron@gmail.com]
Gesendet: Samstag, 26. Mai 2012 01:53
An: solr-user@lucene.apache.org
Betreff: Re: Creating custom Filter / Tokenizer / Request Handler for integration of NER-Framework
Another problem (just discovered this): TokenizerFactories do not get resource handlers. So, you can't go read config or model files for your Tokenizer. TokenFilters do, so you can use the KeywordTokenizer (make one big term) and do your work in a TokenFilter that gets the whole thing.
On Thu, May 24, 2012 at 7:33 AM, Jan Høydahl <ja...@cominvent.com> wrote:
> As Ahmet says, The Update Chain is probably the place to integrate such document oriented processing.
> See http://www.cominvent.com/2011/04/04/solr-architecture-diagram/ for how it integrates with Solr.
>
> --
> Jan Høydahl, search solution architect Cominvent AS -
> www.facebook.com/Cominvent Solr Training - www.solrtraining.com
>
> On 24. mai 2012, at 14:04, Wunderlich, Tobias wrote:
>
>> Hey Guys,
>>
>> I am recently working on a project to integrate a Named-Entity-Recognition-Framework (NER) in an existing searchplatform based on Solr. The Platform uses ManifoldCF to automatically gather the content from various repositories. The NER-Framework creates Annotations/Metadata from given content which I then want to integrate into the search-platform as metadata to use for faceting. Since MCF handles all content gathering, I need a way to integrate the NER-Framework directly into Solr. The Goal is to get all Annotations per document into a multivalued field. My first thought was to create a custom filter, which just takes the content and gives back only the Annotations. But as I understand it, a filter only processes predetermined Tokens, which is useless for my purpose, since the NER-Framework needs to process the whole content of a document. What about a custom Tokenizer? Would it be possible to process the whole text and give back only the Annotations as Tokens? A third thought was to manipulate the ExtractRequestHandler (Solr Cell) used by MCF to somehow add the Annotations as Metadata when the content and metadata is distributed to the different fields.
>>
>> I hope my problem description is sufficient. Does anybody have any thoughts on that subject?
>>
>> Best regards,
>> Tobias
>
--
Lance Norskog
goksron@gmail.com
Re: Creating custom Filter / Tokenizer / Request Handler for
integration of NER-Framework
Posted by Chris Hostetter <ho...@fucit.org>.
: Another problem (just discovered this): TokenizerFactories do not get
: resource handlers. So, you can't go read config or model files for
: your Tokenizer. TokenFilters do, so you can use the KeywordTokenizer
TokenizerFactory subclasses can implement ResourceLoaderAware and load any
resources they want.
-Hoss
Re: Creating custom Filter / Tokenizer / Request Handler for
integration of NER-Framework
Posted by Lance Norskog <go...@gmail.com>.
Another problem (just discovered this): TokenizerFactories do not get
resource handlers. So, you can't go read config or model files for
your Tokenizer. TokenFilters do, so you can use the KeywordTokenizer
(make one big term) and do your work in a TokenFilter that gets the
whole thing.
On Thu, May 24, 2012 at 7:33 AM, Jan Høydahl <ja...@cominvent.com> wrote:
> As Ahmet says, The Update Chain is probably the place to integrate such document oriented processing.
> See http://www.cominvent.com/2011/04/04/solr-architecture-diagram/ for how it integrates with Solr.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.facebook.com/Cominvent
> Solr Training - www.solrtraining.com
>
> On 24. mai 2012, at 14:04, Wunderlich, Tobias wrote:
>
>> Hey Guys,
>>
>> I am recently working on a project to integrate a Named-Entity-Recognition-Framework (NER) in an existing searchplatform based on Solr. The Platform uses ManifoldCF to automatically gather the content from various repositories. The NER-Framework creates Annotations/Metadata from given content which I then want to integrate into the search-platform as metadata to use for faceting. Since MCF handles all content gathering, I need a way to integrate the NER-Framework directly into Solr. The Goal is to get all Annotations per document into a multivalued field. My first thought was to create a custom filter, which just takes the content and gives back only the Annotations. But as I understand it, a filter only processes predetermined Tokens, which is useless for my purpose, since the NER-Framework needs to process the whole content of a document. What about a custom Tokenizer? Would it be possible to process the whole text and give back only the Annotations as Tokens? A third thought was to manipulate the ExtractRequestHandler (Solr Cell) used by MCF to somehow add the Annotations as Metadata when the content and metadata is distributed to the different fields.
>>
>> I hope my problem description is sufficient. Does anybody have any thoughts on that subject?
>>
>> Best regards,
>> Tobias
>
--
Lance Norskog
goksron@gmail.com
Re: Creating custom Filter / Tokenizer / Request Handler for integration of NER-Framework
Posted by Jan Høydahl <ja...@cominvent.com>.
As Ahmet says, The Update Chain is probably the place to integrate such document oriented processing.
See http://www.cominvent.com/2011/04/04/solr-architecture-diagram/ for how it integrates with Solr.
--
Jan Høydahl, search solution architect
Cominvent AS - www.facebook.com/Cominvent
Solr Training - www.solrtraining.com
On 24. mai 2012, at 14:04, Wunderlich, Tobias wrote:
> Hey Guys,
>
> I am recently working on a project to integrate a Named-Entity-Recognition-Framework (NER) in an existing searchplatform based on Solr. The Platform uses ManifoldCF to automatically gather the content from various repositories. The NER-Framework creates Annotations/Metadata from given content which I then want to integrate into the search-platform as metadata to use for faceting. Since MCF handles all content gathering, I need a way to integrate the NER-Framework directly into Solr. The Goal is to get all Annotations per document into a multivalued field. My first thought was to create a custom filter, which just takes the content and gives back only the Annotations. But as I understand it, a filter only processes predetermined Tokens, which is useless for my purpose, since the NER-Framework needs to process the whole content of a document. What about a custom Tokenizer? Would it be possible to process the whole text and give back only the Annotations as Tokens? A third thought was to manipulate the ExtractRequestHandler (Solr Cell) used by MCF to somehow add the Annotations as Metadata when the content and metadata is distributed to the different fields.
>
> I hope my problem description is sufficient. Does anybody have any thoughts on that subject?
>
> Best regards,
> Tobias
Re: Creating custom Filter / Tokenizer / Request Handler for integration of NER-Framework
Posted by Ahmet Arslan <io...@yahoo.com>.
> I am recently working on a project to integrate a
> Named-Entity-Recognition-Framework (NER) in an existing
> searchplatform based on Solr. The Platform uses ManifoldCF
> to automatically gather the content from various
> repositories. The NER-Framework creates Annotations/Metadata
> from given content which I then want to integrate into the
> search-platform as metadata to use for faceting. Since MCF
> handles all content gathering, I need a way to integrate the
> NER-Framework directly into Solr. The Goal is to get all
> Annotations per document into a multivalued field. My
> first thought was to create a custom filter, which just
> takes the content and gives back only the Annotations.
> But as I understand it, a filter only processes
> predetermined Tokens, which is useless for my purpose, since
> the NER-Framework needs to process the whole content of a
> document. What about a custom Tokenizer? Would it be
> possible to process the whole text and give back only the
> Annotations as Tokens? A third thought was to manipulate the
> ExtractRequestHandler (Solr Cell) used by MCF to somehow add
> the Annotations as Metadata when the content and metadata is
> distributed to the different fields.
>
> I hope my problem description is sufficient. Does anybody
> have any thoughts on that subject?
UpdateRequestProcessor is more appropriate in this case. Like http://wiki.apache.org/solr/SolrUIMA