You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by ianwong <yi...@hotmail.com> on 2008/12/18 14:11:04 UTC

Re: distributed search servers

Dear Dennis,

about your answer|:

> 2) does tomcat wait for ALL search servers to respond before displaying
> the query result or does it display the result as soon as one server
> responds?

Yes, to a timeout value.  If one goes down it will slow down the entire 
search cluster.

how can I change the timeout value? what is the default value?

Ian




Dennis Kubes-2 wrote:
> 
> 
> 
> Hilkiah Lavinier wrote:
>> Hi all,
>> 
>> Have a distributed search issue I need some advice on.  The scenario is
>> that I have tomcat running off one server and two nutch search servers
>> running off two other machines (so 3 machines in total).  I've setup the
>> nutch war to correctly call the search servers and they respond.  Problem
>> is I get duplicate results.  Now I have the same data/information from
>> the crawl copied on both machines so the crawl data is replicated on both
>> machines.
>> 
>> Questions:
>> 1) how do I prevent the duplicate response? If I start a third search
>> server I only get two duplicate responses so it doesn't seem to increase
>> with the number of search servers
> 
> In your query or in NutchBean set the hitsPerSite=1, here is an example:
> 
> Duplicates:
> http://search.isc.swlabs.org/search.jsp?lang=en&query=java
> 
> No Duplicates:
> http://search.isc.swlabs.org/search.jsp?lang=en&query=java&hitsPerSite=1
> 
> This is based on hostname so for instance java.net and www.java.net will 
> be considered different even though they are the same.  The latter 
> problem has not been corrected yet in Nutch, but we are working on it.
> 
>> 2) does tomcat wait for ALL search servers to respond before displaying
>> the query result or does it display the result as soon as one server
>> responds?
> 
> Yes, to a timeout value.  If one goes down it will slow down the entire 
> search cluster.
> 
>> 3) in terms of load sharing, what is the best approach for distributed
>> search servers?
> 
> If you are looking at a round-robin sort of load balancing I would say 
> two nutch servers hitting different search servers with replicated 
> content fronted by an apache server or hardware load balancer.  Remember 
> that the entire search can still be up even if one or more search 
> servers fail.  I would worry more about clustering the front end search 
> website than load balancing the search servers but it all depends on 
> what your goal is.  For a www search we don't care if a few of the 
> search servers are down as long as the search is functional.
> 
> Dennis Kubes
> 
> 
>> 
>> Any help would be greatly appreciated!
>> 
>> Thanks,
>> 
>> Hilkiah G. Lavinier MEng (Hons), ACGI 
>> 6 Winston Lane, 
>> Goodwill, 
>> Roseau, Dominica 
>> Mbl: (767) 275 3382
>> Hm : (767) 440 3924
>> Fax: (767) 440 4991
>> VoIP USA: (646) 432 4487
>>  
>> Email: hilkiah@yahoo.com
>> Email: hilkiah.lavinier@gmail.com
>> IM: Yahoo hilkiah / MSN hilkiahlavinier@hotmail.com
>> IM: ICQ #8978201  / AOL hilkiah21
>> 
>> 
>> 
>> 
>> 
>>      
>> ____________________________________________________________________________________
>> Looking for last minute shopping deals?  
>> Find them fast with Yahoo! Search. 
>> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
> 
> 

-- 
View this message in context: http://www.nabble.com/distributed-search-servers-tp14975657p21072494.html
Sent from the Nutch - User mailing list archive at Nabble.com.