You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Spadez <ja...@hotmail.com> on 2012/04/16 13:55:27 UTC
Collecting Entries in Nutch - Moving to Solr
Hi.
I beleive it is possible to move data collected from Nutch straight to Solr
to make it searchable, i.e this structure:
Nutch -> Solr
However, I need to do a bit of post processing on this data. One example is
changing the scrapped city name to Long and Lat co-ordinates. Therefore my
option as far as I can see it is this:
Nutch -> MySQL -> Post Processing -> Solr
I was wondering what peoples opinions on this where. We are looking at
around 10,000 new entries a day with around 300,000 entries at any one time.
Is there any better way of doing it? Is this kind of setup used a lot?
Any helpful information or advice on this would be really appreciated.
Regards,
James
--
View this message in context: http://lucene.472066.n3.nabble.com/Collecting-Entries-in-Nutch-Moving-to-Solr-tp3914183p3914183.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Collecting Entries in Nutch - Moving to Solr
Posted by Julien Nioche <li...@gmail.com>.
Hi James
You could write a custom IndexingFilter to generate new fields for the Long
/ Lat coordinates from the city name you'll have extracted and add the
corresponding fields in the SOLR schema. You can then use the SOLR indexing
in Nutch as usual.
Another option is to use Behemoth
*https://github.com/jnioche/behemoth*<https://github.com/jnioche/behemoth>to
import the Nutch segments, write your own module for extracting the
geolocations then use the SOLR module.
HTH
Julien
On 16 April 2012 12:55, Spadez <ja...@hotmail.com> wrote:
> Hi.
>
> I beleive it is possible to move data collected from Nutch straight to
> Solr
> to make it searchable, i.e this structure:
>
> Nutch -> Solr
>
> However, I need to do a bit of post processing on this data. One example is
> changing the scrapped city name to Long and Lat co-ordinates. Therefore my
> option as far as I can see it is this:
>
> Nutch -> MySQL -> Post Processing -> Solr
>
> I was wondering what peoples opinions on this where. We are looking at
> around 10,000 new entries a day with around 300,000 entries at any one
> time.
> Is there any better way of doing it? Is this kind of setup used a lot?
>
> Any helpful information or advice on this would be really appreciated.
>
> Regards,
>
> James
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collecting-Entries-in-Nutch-Moving-to-Solr-tp3914183p3914183.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble