[c++ cluster] Persistent cluster restart design

[Alan Conway <ac...@redhat.com>]

Some initial rough ideas on how to handle persistent cluster restart - how to 
choose the right database when restarting a cluster that has been fully stopped. 
Suggestions appreciated.

* Persistent cluster restart

** Requirements

Identify the "good" database on restart of persistent cluster. Update
all members from the same database. Possibly allow multiple members
with the same good database to start from their database.

Catch gross errors, e.g. trying to include a broker with an unrelated
database in the cluster.

Start brokers in any order: Brokers will normally be started via boot
scripts so it may not be easy for user to control the start-up
sequence.  Any start-up-ordering should be negotiated among the
brokers automatically. Each broker holds off client connections until
the cluster is started.

Orderly shut down: Some members die or are killed but not all.
Multiple surviving members are stopped with cluster-qpid -k, or the
sole survivor is stopped with qpidd -q/kill -TERM.  On resart the
survivors can use their own database, others must wait for an update.

Total failure: all members die, there are no survivors. On restart we
use the most up-to-date database if it can be determined. If not we
fail with an informative log message about how to manually resolve the
situation.

*Mixing persistent & non-persistent members?*

** Implementation

New option, required for persistent cluster members:

--cluster-start-count N: Members a cluster will wait till N members
have joined before handling client requests. Persistence negotiations
occur when all N members are present.

Consistency check: if any member has --cluster-start-count then all
must have the same value.

*Relaxed once start-up complete - allow more newcomers?*

Enforcing consistency rules: have to shut down cluster? what status to
record for next start-up attempt? Manual intervention?

Backwards compatibility: without --cluster-start-count we do the
existing start-up thing, i.e. move the store out of the way except for
first member.

*** Stored state

At each config change we record: cluster name, membership snapshot, config seq, 
frame seq.
  - frame seq = cluster wide frame sequence number 64 bits
  - config seq = cluster wide config-change count (do we need it as well as 
frame seq?)

At qpid-cluster -k or any shutdown if we are last in cluster we record a "clean 
store" marker.

*maybe update frameseq at some interval, same for all cluster members to help 
with total failure?*

*** Startup

Wait for N members

Members with clean stores:
  - compare stored state with other clean store members.
  - load their own store, if any mismatch then manual intervention
  - provide updates to members without a clean store.

No member with clean store: member with newest state loads store and
provides updates.

Tie for newest state: manual intervention.

Errors during startup: members log critical errors but do not shut down, so they
can inform any new members that the cluster is broken. Need to use cluster-qpid -k
to shut down and do manual intervention. *TODO manual steps*

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org