Working with Solr. Doubts

[Javier Puerto <ja...@juntadeandalucia.es>]

Hi to all,

We are working with Nutch 0.8 to crawl about 18 web sites in an
intranet, each site has an average of 40.000~50.000 documents. Actually
we had the contents splits in four parts, and ran a
DistributedSearchServer for each and the client configured for the 4
servers.

Now we had a Apache Droids crawling the filesystem for transform and
index a lot of documents to Solr. we need to unify the front-end client
to be able to search in Solr and Nutch.


I thought to upgrade the release of Nutch for the Solr support but I
have some doubts:

Nutch has a front-end for Solr or I had to develop all?
Can I search on multiple Solr servers in the same way that Nutch with
DistributedSearchServer?
Can I search on Nutch and Solr simultaneously and merge the results?

If anyone has had any similar problem or any suggestions to clarify my
doubts, thank you very much!

Salu2.

Re: Working with Solr. Doubts

[Doğacan Güney <do...@gmail.com>]

Hi,

On Mon, Mar 9, 2009 at 20:08, Javier Puerto
<ja...@juntadeandalucia.es> wrote:
> Hi to all,
>
> We are working with Nutch 0.8 to crawl about 18 web sites in an
> intranet, each site has an average of 40.000~50.000 documents. Actually
> we had the contents splits in four parts, and ran a
> DistributedSearchServer for each and the client configured for the 4
> servers.
>
> Now we had a Apache Droids crawling the filesystem for transform and
> index a lot of documents to Solr. we need to unify the front-end client
> to be able to search in Solr and Nutch.
>
>
> I thought to upgrade the release of Nutch for the Solr support but I
> have some doubts:
>
> Nutch has a front-end for Solr or I had to develop all?

Yes, 1.0 release will have this feature.

> Can I search on multiple Solr servers in the same way that Nutch with
> DistributedSearchServer?

You can search on multiple solr servers but nutch does not provide
any support if you want to index to multiple servers. Indexing looks like this:

bin/nutch solrindex <solr url> <crawldb> <linkdb> <segments>

So you can do something like this (I think, never tested it but it
should work):

bin/nutch solrindex <solr1 url> <crawldb> <linkdb> <segment1>
bin/nutch solrindex <solr2 url> <crawldb> <linkdb> <segment2>
......

> Can I search on Nutch and Solr simultaneously and merge the results?
>

Yes.

> If anyone has had any similar problem or any suggestions to clarify my
> doubts, thank you very much!
>
> Salu2.
>
>



-- 
Doğacan Güney

Re: Working with Solr. Doubts

[Sami Siren <ss...@gmail.com>]

Javier Puerto wrote:
> Hi to all,
>
> We are working with Nutch 0.8 to crawl about 18 web sites in an
> intranet, each site has an average of 40.000~50.000 documents. Actually
> we had the contents splits in four parts, and ran a
> DistributedSearchServer for each and the client configured for the 4
> servers.
>
> Now we had a Apache Droids crawling the filesystem for transform and
> index a lot of documents to Solr. we need to unify the front-end client
> to be able to search in Solr and Nutch.
>
>
> I thought to upgrade the release of Nutch for the Solr support but I
> have some doubts:
>
> Nutch has a front-end for Solr or I had to develop all?
> Can I search on multiple Solr servers in the same way that Nutch with
> DistributedSearchServer?
> Can I search on Nutch and Solr simultaneously and merge the results?
>
> If anyone has had any similar problem or any suggestions to clarify my
> doubts, thank you very much!
>   

I think you could simplify your setup by using solr, scale should not be 
problem, last week I tested the Solr Nutch integration with collection 
size of over 6 M docs on an old pc (growing roughly 1 M per day). 
Response times were still pretty good for simple queries even when I let 
Solr create snippets.

--
 Sami Siren