High Availability

[Matthew Inger <ma...@yahoo.com>]

I'm kind of stuck and looking for suggestions for high availability options.  I've figured out without much trouble how to get the master-slave replication working.  This eliminates any single points of failure in the application in terms of the application's searching capability.

I would setup a master which would create the index and several slaves to act as the search servers, and put them behind a load balancer to distribute the requests.  This would ensure that if a slave node goes down, requests would continue to get serviced by the other nodes that are still up.

The problem I have is that my particular application also has the capability to trigger index updates from the user interface.  This means that the master now becomes a single point of failure for the user interface.  

The basic idea of the app is that there are multiple oracle instances contributing to a single document.  The volume and organization of the data (database links, normalization, etc...) prevents any sort of fast querying via SQL to do querying of the documents.  The solution is to build a lucene index (via solr), and use that for searching.  When updates are made in the UI, we will also send the updates directly to the solr server as well (we don't want to wait some arbitrary interval for a delta query to run).  

So you can see the problem here is that if the master is down, the sending of the updates to the master solr server will fail, thus causing an application exception.

I have tried configuring multiple solr servers which are both setup as masters and slaves to each other, but they keep clobber each other's index updates and rolling back each other's delta updates.  It seems that the replication doesn't take the generation # into account and check that the generation it's fetching is > the generation it already has before it applies it.

I thought of maybe introducing a JMS queue to send my updates to and having the JMS message listener set to manually acknowledge the messages only after a succesfull application of the solrj api calls, but that seems kind of contrived, and is only a band-aid.

Does anyone have any suggestions?


 ----
mattinger@yahoo.com
"Once you start down the dark path, forever will it
dominate your destiny.  Consume you it will " - Yoda