Solr-303 Re: Solr in a distributed multi-machine high-performance environment

[Srikant Jakilinki <sr...@bluebottle.com>]

Yes, 303 looks very promising. And I would like to get involved. I have 
gone to the JIRA thread and very impressed by the activity going on 
there. It is THE hangout :-)

Following up, does anyone (especially Yonik or Sharad) have any 
documentation of this feature? Such as goals, use cases, requirements, 
design, implementation spcification, task lists, install manual etc. 
Just about anything at all... Even text notes would do as I intend to 
write it up properly.

If so, please pass them over by replying to my ID. I want to make Solr 
work in such a way OOB that when one sets it up to work on a bunch of 
machines (one config file only need be touched), the 'distributed 
cluster' - herafter, referred to as a 'node' - should be able to handle 
an index the size of the RAMs combined for the fastest response possible 
with near real-time reflexes of index/search/results. The maximum 
storage would be the size of the HDDs combined but comes at a penalty of 
responsiveness which is nothing but classic space-time tradeoff.
The user should be able to set a response limit which when hit 
(autonomic principle?), the node should alert the user to add more RAM 
or machines. The node should be fault-tolerant and adding a machine to 
the node should make the system scale linearly. Yada yada yada...
Solr is a black box for search. Solr 'node' should be so too.

Of course, any requests can be sent too and I will incorporate in the 
documentation.

So, guys, any documentaion on Solr-303 please,
Srikant

Shalin Shekhar Mangar wrote:
> Look at http://issues.apache.org/jira/browse/SOLR-303
>
> Please note that it is still work in progress. So you may not be able to use
> it immeadiately.
>
> On Jan 16, 2008 10:53 AM, Srikant Jakilinki <sr...@gmail.com> wrote:
>
>   
>> Hi All,
>>
>> There is a requirement in our group of indexing and searching several
>> millions of documents (TREC) in real-time and millisecond responses.
>> For the moment we are preferring scale-out (throw more commodity
>> machines) approaches rather than scale-up (faster disks, more
>> RAM). This is in-turn inspired by the "Scale-out vs. Scale-up" paper
>> (mail me if you want a copy) in which it was proven that this kind of
>> distribution scales better and is more resilient.
>>
>> So, are there any resources available (Wiki, Tutorials, Slides, README
>> etc.) that throw light and guide newbies on how to run Solr in a
>> multi-machine scenario? I have gone through the mailing lists and site
>> but could not really find any answers or hands-on stuff to do so. An
>> adhoc guideline to get things working with 2 machines might just be
>> enough but for the sake of thinking out loud and solicit responses
>> from the list, here are my questions:
>>
>> 1) Solr that has to handle a fairly large index which has to be split
>> up on multiple disks (using Multicore?)
>> - Space is not a problem since we can use NFS but that is not
>> recommended as we would only exploit 1 processor
>> 2) Solr that has to handle a large collective index which has to be
>> split up on multi-machines
>> - The index is ever increasing (TB scale) and dynamic and all of it
>> has to be searched at any point
>> 3) Solr that has to exploit multi-machines because we have plenty of
>> them in a tightly coupled P2P scenario
>> - Machines are not a problem but will they be if they are of varied
>> configurations (PIII to Core2; Linux to Vista; 32-bit to 64-bit; J2SE
>> 1.1 to 1.6)
>> 4) Solr that has to distribute load on several machines
>> - The index(s) could be common though like say using a distributed
>> filesystem (Hadoop?)
>>
>> In each the above cases (we might use all of these strategies at
>> various use cases) the application should use Solr as a strict backend
>> and named service (IP or host:port) so that we can expose this
>> application (and the service) to the web or intranet. Machine failures
>> should be tolerated too. Also, does Solr manage load balancing out of
>> the box if it was indeed configured to work with multi-machines?
>>
>> Maybe it is superfluous but is Solr and/or Nutch the only way to use
>> Lucene in a multi-machine environment? Or is there some hidden
>> document/project somewhere that makes it possible by exposing a
>> regular Lucene process over the network using RMI or something? It is
>> my understanding (could be wrong) that Nutch and to some extent, Solr
>> do not perform well when there is a lot of indexing activity in
>> parallel to search. Batch processing is also there and perhaps we can
>> use Nutch/Solr there. Even so, we need multi-machine directions.
>>
>> I am sure that multi-machines make possible for a lot of other ways
>> which might solve the goal better and that others have practical
>> experience on. So, any advise and tips are also very welcome. We
>> intend to document things and do some benchmarking along the way in
>> the open spirit.
>>
>> Really sorry for the length but I hope some answers are forthcoming.
>>
>> Cheers,
>> Srikant
>>
>>     
>
>
>
>

----------------------------------------------------------------------
Get a free email account with anti spam protection.
http://www.bluebottle.com/tag/2