You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vineet yadav <vi...@gmail.com> on 2015/01/30 15:07:22 UTC
Calling custom request handler with data import
Hi,
I am using data import handler to import data from mysql, and I want to
identify name entities from it. So I am using following example(
http://www.searchbox.com/named-entity-recognition-ner-in-solr/). where I am
using stanford ner to identify name entities. I am using following
requesthandler
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-import.xml</str>
</lst>
</requestHandler>
for importing data from mysql and
<requestHandler name="/ner" class="com.searchbox.ner.NerHandler" />
<updateRequestProcessorChain name="mychain" >
<processor class="com.searchbox.ner.NerProcessorFactory" >
<lst name="queryFields">
<str name="queryField">content</str>
</lst>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="update.chain">mychain</str>
</lst>
</requestHandler>
for identifying name entities.NER request handler identifies name entities
from content field, but store extracted entities in solr fields.
NER request handler was working when I am using nutch with solr. But When I
am importing data from mysql, ner request handler is not invoked. So
entities are not stored in solr for imported documents. Can anybody tell me
how to call custom request handler in data import handler.
Otherwise if I can invoke ner request handler externally, so that it can
index person, organization and location in solr for imported document. It
is also fine. Any suggestion are welcome.
Thanks
Vineet Yadav
Re: Calling custom request handler with data import
Posted by Dan Davis <da...@gmail.com>.
You know, another thing you can do is just write some Java/perl/whatever to
pull data out of your database and push it to Solr. Not as convenient
for development perhaps, but it has more legs in the long run. Data
Import Handler does not easily multi-thread.
On Sat, Jan 31, 2015 at 12:34 AM, Dan Davis <da...@gmail.com> wrote:
> The Data Import Handler isn't pushing data into the /update request
> handler. However, Data Import Handler can be extended with transformers.
> Two such transformers are the TemplateTransformer and the
> ScriptTransformer. It may be possible to get a script function to load
> your custom Java code. You could also just write a
> StandfordNerTransformer.
>
> Hope this helps,
>
> Dan
>
> On Fri, Jan 30, 2015 at 9:07 AM, vineet yadav <vineet.yadav.iiit@gmail.com
> > wrote:
>
>> Hi,
>> I am using data import handler to import data from mysql, and I want to
>> identify name entities from it. So I am using following example(
>> http://www.searchbox.com/named-entity-recognition-ner-in-solr/). where I
>> am
>> using stanford ner to identify name entities. I am using following
>> requesthandler
>>
>> <requestHandler name="/dataimport"
>> class="org.apache.solr.handler.dataimport.DataImportHandler">
>> <lst name="defaults">
>> <str name="config">data-import.xml</str>
>> </lst>
>> </requestHandler>
>>
>> for importing data from mysql and
>>
>> <requestHandler name="/ner" class="com.searchbox.ner.NerHandler" />
>> <updateRequestProcessorChain name="mychain" >
>> <processor class="com.searchbox.ner.NerProcessorFactory" >
>> <lst name="queryFields">
>> <str name="queryField">content</str>
>> </lst>
>> </processor>
>> <processor class="solr.LogUpdateProcessorFactory" />
>> <processor class="solr.RunUpdateProcessorFactory" />
>> </updateRequestProcessorChain>
>> <requestHandler name="/update" class="solr.UpdateRequestHandler">
>> <lst name="defaults">
>> <str name="update.chain">mychain</str>
>> </lst>
>> </requestHandler>
>>
>> for identifying name entities.NER request handler identifies name entities
>> from content field, but store extracted entities in solr fields.
>>
>> NER request handler was working when I am using nutch with solr. But When
>> I
>> am importing data from mysql, ner request handler is not invoked. So
>> entities are not stored in solr for imported documents. Can anybody tell
>> me
>> how to call custom request handler in data import handler.
>>
>> Otherwise if I can invoke ner request handler externally, so that it can
>> index person, organization and location in solr for imported document. It
>> is also fine. Any suggestion are welcome.
>>
>> Thanks
>> Vineet Yadav
>>
>
>
Re: Calling custom request handler with data import
Posted by Dan Davis <da...@gmail.com>.
The Data Import Handler isn't pushing data into the /update request
handler. However, Data Import Handler can be extended with transformers.
Two such transformers are the TemplateTransformer and the
ScriptTransformer. It may be possible to get a script function to load
your custom Java code. You could also just write a
StandfordNerTransformer.
Hope this helps,
Dan
On Fri, Jan 30, 2015 at 9:07 AM, vineet yadav <vi...@gmail.com>
wrote:
> Hi,
> I am using data import handler to import data from mysql, and I want to
> identify name entities from it. So I am using following example(
> http://www.searchbox.com/named-entity-recognition-ner-in-solr/). where I
> am
> using stanford ner to identify name entities. I am using following
> requesthandler
>
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
> <lst name="defaults">
> <str name="config">data-import.xml</str>
> </lst>
> </requestHandler>
>
> for importing data from mysql and
>
> <requestHandler name="/ner" class="com.searchbox.ner.NerHandler" />
> <updateRequestProcessorChain name="mychain" >
> <processor class="com.searchbox.ner.NerProcessorFactory" >
> <lst name="queryFields">
> <str name="queryField">content</str>
> </lst>
> </processor>
> <processor class="solr.LogUpdateProcessorFactory" />
> <processor class="solr.RunUpdateProcessorFactory" />
> </updateRequestProcessorChain>
> <requestHandler name="/update" class="solr.UpdateRequestHandler">
> <lst name="defaults">
> <str name="update.chain">mychain</str>
> </lst>
> </requestHandler>
>
> for identifying name entities.NER request handler identifies name entities
> from content field, but store extracted entities in solr fields.
>
> NER request handler was working when I am using nutch with solr. But When I
> am importing data from mysql, ner request handler is not invoked. So
> entities are not stored in solr for imported documents. Can anybody tell me
> how to call custom request handler in data import handler.
>
> Otherwise if I can invoke ner request handler externally, so that it can
> index person, organization and location in solr for imported document. It
> is also fine. Any suggestion are welcome.
>
> Thanks
> Vineet Yadav
>
Re: Calling custom request handler with data import
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
at your service!
On Sat, Jan 31, 2015 at 1:00 PM, vineet yadav <vi...@gmail.com>
wrote:
> Hi mikhail,
> Thanks for the suggestion. it is helpful.
>
> Regards
> Vineet Yadav
>
>
> On Sat, Jan 31, 2015 at 2:38 PM, Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
>
> > Did you try to specify update processor?, ie
> >
> > On Fri, Jan 30, 2015 at 5:07 PM, vineet yadav <
> vineet.yadav.iiit@gmail.com
> > >
> > wrote:
> >
> > > <requestHandler name="/dataimport"
> > > class="org.apache.solr.handler.dataimport.DataImportHandler">
> > > <lst name="defaults">
> > > <str name="config">data-import.xml</str>
> > >
> >
> > <str name="update.chain">/ner</str>
> >
> > </lst>
> > > </requestHandler>
> > >
> >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
> >
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Calling custom request handler with data import
Posted by vineet yadav <vi...@gmail.com>.
Hi mikhail,
Thanks for the suggestion. it is helpful.
Regards
Vineet Yadav
On Sat, Jan 31, 2015 at 2:38 PM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Did you try to specify update processor?, ie
>
> On Fri, Jan 30, 2015 at 5:07 PM, vineet yadav <vineet.yadav.iiit@gmail.com
> >
> wrote:
>
> > <requestHandler name="/dataimport"
> > class="org.apache.solr.handler.dataimport.DataImportHandler">
> > <lst name="defaults">
> > <str name="config">data-import.xml</str>
> >
>
> <str name="update.chain">/ner</str>
>
> </lst>
> > </requestHandler>
> >
>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: Calling custom request handler with data import
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Did you try to specify update processor?, ie
On Fri, Jan 30, 2015 at 5:07 PM, vineet yadav <vi...@gmail.com>
wrote:
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
> <lst name="defaults">
> <str name="config">data-import.xml</str>
>
<str name="update.chain">/ner</str>
</lst>
> </requestHandler>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>