Questions about failover in master / slave configuration

[Laurent Moret <mo...@gmail.com>]

Hi

Our team plans to use SolR for our searchs (we currently use Lucene
directly, but some new business requirements such as "near real time" index
updates seems to be easier to handle with solr).

Our application runs with 2 data centers in active / active mode.
Read operations in the index are made by web or web services applications,
write operations by batch programs.

We studied the documentation and wiki on the replication mechanisms, but we
have some questions about the way it will work...

What we plan :
- In data center A (where batch programs updating the index run in "normal"
behaviour) : a master solr server M1 (updates will be done to this one), and
a slave S1 replicating from M1 queried by applications from this data center
- In data center B : a pseudo master M2 configured to synchronise from M1,
and as a repeater to dispatch updates to this data center, and a slave S2
replicating from M2 queried by applications from this data center

Our questions are about failover management :
- If a slave is down in a datacenter, we can use a standard HTTP failover /
load balancing system to use the slave from the other data center for search
operations from applications (if S1 is down for example, search operations
from application in both data centers A and B will use S2), this is the
simple point
- If M2 is down, we have to update solr config on S2 so it replicates
directly from M1 instead of using the M2 repeater. How can we do this
without restarting S2 ?
- And the most complicated : M1 is down... In this case, M2 can become the
"new master". But to do so, we have to update end point for batch programs
that update (so they push updates to M2 instead of M1), update solr config
in M2 so it doesn't try to synchronize from M1, and update solr config in S1
so it uses M2 as master server... And what will happen when M1 is back ? Its
index will be older than the one in M2, so it has to synch from it...

We don't have any experience with solr, and this must be a common problem.
Maybe someone can help and give us some tips ?

Thanks !

Laurent