Scaling up to several machines with Lucene

[Chun Wei Ho <cw...@gmail.com>]

Hi,

We are currently running a Tomcat web application serving searches
over our Lucene index (10GB) on a single server machine (Dual 3GHz
CPU, 4GB RAM). Due to performance issues and to scale up to handle
more traffic/search requests, we are getting another server machine.

We are looking at two ways of scaling:
(1) duplicating the web application and index on the second machine
and load-balancing incoming users between the two servers.

(2) modifying our web application so that one machine will host our
web application (and associated MySQL database), while the other one
will host the Lucene index. The first machine would be dedicated to
our web application and database, while the second becomes our
dedicated Lucene search server. When users perform a search on the
website, the web application will send the request to the Lucene index
server, which will perform the search and return the results to the
web application.

We would like comments from users who have set up similar systems on
how you have accomplished (1) in your setups, and whether (2) is a
good choice for scaling.


Attached is a more complete RTF document outlining our architecture
and proposal. We appreciate your perusal and comments.

Regards,
CW