You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Mambe Churchill Nanje <ma...@afrovisiongroup.com> on 2010/03/20 15:27:07 UTC
Nutch for crawling and indexing with solr
I would like to know if anybody has used nutch for crawling and indexing
with Solr
I am currently doing intranet crawling with the nutch/bin crawl
functionality and I use nutch/bin solrindex function to push the nutch
segments, linkdb, crawldb to solr index.
I am wondering if I delete the crawl folder and do a full crawl at a new
date, if it gets the new content and push to solr index, will this help me
get only recent documents added to the solr ending, I want to be getting
only recent documents indexed. So i was thinking that nutch crawl, and then
push to solr will only add the new documents but i am not sure because I
dont have experience.
Can someone tell me if this will work, because I have been searching for
re-crawl for nutch for like ever and I think nutch crawl and solr index over
and over can be like a work around for recrawling
thanks
Mambe Churchill Nanje
237 77545907,
AfroVisioN Founder, President,CEO
http://www.afrovisiongroup.com
http://mambenanje.blogspot.com
skypeID: mambenanje
Re: Nutch for crawling and indexing with solr
Posted by Mambe Churchill Nanje <ma...@afrovisiongroup.com>.
ooh thanks for the help
will check that out and also do you have any knowledge how solr indexing
works. because I want it to take care of adding new documents and rejecting
old one and not updating their date. so that i can use that date to fetch
results with date range
thanks
Mambe Churchill Nanje
237 77545907,
AfroVisioN Founder, President,CEO
http://www.afrovisiongroup.com
http://mambenanje.blogspot.com
skypeID: mambenanje
On Sun, Mar 21, 2010 at 4:47 AM, Hannes Carl Meyer <
hannescarl@googlemail.com> wrote:
> Hi Mambe,
> you should check out this article from Lucid Imagination:
> http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/
> Regards,
> Hannes
>
> http://de.linkedin.com/in/hannescarlmeyer
> http://www.xing.com/profile/HannesCarl_Meyer
>
> On Sat, Mar 20, 2010 at 3:27 PM, Mambe Churchill Nanje <
> mambenanje@afrovisiongroup.com> wrote:
>
> > I would like to know if anybody has used nutch for crawling and indexing
> > with Solr
> > I am currently doing intranet crawling with the nutch/bin crawl
> > functionality and I use nutch/bin solrindex function to push the nutch
> > segments, linkdb, crawldb to solr index.
> > I am wondering if I delete the crawl folder and do a full crawl at a new
> > date, if it gets the new content and push to solr index, will this help
> me
> > get only recent documents added to the solr ending, I want to be getting
> > only recent documents indexed. So i was thinking that nutch crawl, and
> then
> > push to solr will only add the new documents but i am not sure because I
> > dont have experience.
> > Can someone tell me if this will work, because I have been searching for
> > re-crawl for nutch for like ever and I think nutch crawl and solr index
> > over
> > and over can be like a work around for recrawling
> >
> > thanks
> >
> > Mambe Churchill Nanje
> > 237 77545907,
> > AfroVisioN Founder, President,CEO
> > http://www.afrovisiongroup.com
> > http://mambenanje.blogspot.com
> > skypeID: mambenanje
> >
>
Re: Nutch for crawling and indexing with solr
Posted by Hannes Carl Meyer <ha...@googlemail.com>.
Hi Mambe,
you should check out this article from Lucid Imagination:
http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/
Regards,
Hannes
http://de.linkedin.com/in/hannescarlmeyer
http://www.xing.com/profile/HannesCarl_Meyer
On Sat, Mar 20, 2010 at 3:27 PM, Mambe Churchill Nanje <
mambenanje@afrovisiongroup.com> wrote:
> I would like to know if anybody has used nutch for crawling and indexing
> with Solr
> I am currently doing intranet crawling with the nutch/bin crawl
> functionality and I use nutch/bin solrindex function to push the nutch
> segments, linkdb, crawldb to solr index.
> I am wondering if I delete the crawl folder and do a full crawl at a new
> date, if it gets the new content and push to solr index, will this help me
> get only recent documents added to the solr ending, I want to be getting
> only recent documents indexed. So i was thinking that nutch crawl, and then
> push to solr will only add the new documents but i am not sure because I
> dont have experience.
> Can someone tell me if this will work, because I have been searching for
> re-crawl for nutch for like ever and I think nutch crawl and solr index
> over
> and over can be like a work around for recrawling
>
> thanks
>
> Mambe Churchill Nanje
> 237 77545907,
> AfroVisioN Founder, President,CEO
> http://www.afrovisiongroup.com
> http://mambenanje.blogspot.com
> skypeID: mambenanje
>