You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@qpid.apache.org by ac...@apache.org on 2011/10/12 21:55:34 UTC
svn commit: r1182558 - in
/qpid/branches/qpid-2920-active/qpid/cpp/design_docs:
new-cluster-design.txt new-cluster-plan.txt
Author: aconway
Date: Wed Oct 12 19:55:34 2011
New Revision: 1182558
URL: http://svn.apache.org/viewvc?rev=1182558&view=rev
Log:
QPID-2920: Update new-cluster-plan.txt and new-cluster-design.txt.
Filled out outstanding tasks in plan.
Added comments on live upgrades to design and plan.
Modified:
qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-design.txt
qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-plan.txt
Modified: qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-design.txt
URL: http://svn.apache.org/viewvc/qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-design.txt?rev=1182558&r1=1182557&r2=1182558&view=diff
==============================================================================
--- qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-design.txt (original)
+++ qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-design.txt Wed Oct 12 19:55:34 2011
@@ -234,10 +234,10 @@ Note:
- Updatee stalls clients until the update completes.
(Note: May be possible to avoid updatee stall as well, needs thought)
-** Cluster API
+** Internal cluster interface
-The new cluster API is similar to the MessageStore interface, but
-provides more detail (message positions) qand some additional call
+The new cluster interface is similar to the MessageStore interface, but
+provides more detail (message positions) and some additional call
points (e.g. acquire)
The cluster interface captures these events:
@@ -284,13 +284,33 @@ cluster uses the same messagea allocatio
standalone broker, with a little extra asynchronous book-keeping.
If a queue has multiple consumers connected to multiple brokers, the
-new cluster time shares the queue which is less efficient than having
-all consumers connected to the same broker.
+new cluster time-shares the queue which is less efficient than having
+all consumers on a queue connected to the same broker.
** Flow control
New design does not queue up CPG delivered messages, they are
processed immediately in the CPG deliver thread. This means that CPG's
- flow control is sufficient for qpid.
+flow control is sufficient for qpid.
+
+** Live upgrades
+
+Live upgrades refers to the ability to upgrade a cluster while it is
+running, with no downtime. Each brokers in the cluster is shut down,
+and then re-started with a new version of the broker code.
+
+To achieve this
+- Cluster protocl XML file has a new element <version number=N> attached
+ to each method. This is the version at which the method was added.
+- New versions can only add methods, existing methods cannot be changed.
+- The cluster handshake for new members includes the protocol version
+ at each member.
+- The cluster's version is the lowest version among its members.
+- A newer broker can join and older cluster. When it does, it must restrict
+ itself to speaking the older version protocol.
+- When the cluster version increases (because the lowest version member has left)
+ the remaining members may move up to the new version.
+
+
* Design debates
** Active/active vs. active passive
Modified: qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-plan.txt
URL: http://svn.apache.org/viewvc/qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-plan.txt?rev=1182558&r1=1182557&r2=1182558&view=diff
==============================================================================
--- qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-plan.txt (original)
+++ qpid/branches/qpid-2920-active/qpid/cpp/design_docs/new-cluster-plan.txt Wed Oct 12 19:55:34 2011
@@ -25,8 +25,8 @@ Meaning of priorities:
[#C] Can be addressed in a later release.
The existig prototype is bare bones to do performance benchmarks:
-- Implement publish and consumer locking protocol.
-- Defered delivery and asynchronous completion of message till self-delivered.
+- Implements publish and consumer locking protocol.
+- Defered delivery and asynchronous completion of message.
- Optimize the case all consumers are on the same node.
- No new member updates, no failover updates, no transactions, no persistence etc.
@@ -79,7 +79,6 @@ Multicast using fixed-size (64k) buffers
* Open questions
-
** TODO [#A] Queue sequence numbers vs. independant message IDs.
SCHEDULED: <2011-10-07 Fri>
@@ -94,18 +93,18 @@ However they imply additional map lookup
- [ ] Prototype independent message IDs, check performance.
* Outstanding Tasks
-** TODO [#A] Defer and async complete wiring commands.
+** TODO [#A] Defer and async completion of wiring commands.
Testing requirement: Many tests assume wiring changes are visible
across the cluster once the commad completes.
-Name clashes: avoid race if same name queue/exchange declared on 2
-brokers simultaneously
+Name clashes: need to avoid race if same name queue/exchange declared
+on 2 brokers simultaneously
-** TODO [#B] Management support
+** TODO [#A] Passing all existing cluster tests.
-- Replicate management methods that modify queues - e.g. move, purge.
-- Report connections - local only or cluster-wide?
+The new cluster should be a drop-in replacement for the old, so it
+should be able to pass all the existing tests.
** TODO [#A] Update to new members joining.
@@ -140,29 +139,28 @@ Exchange updatee:
Updater remains active throughout.
Updatee stalls clients until the update completes.
-** TODO [#B] TX transaction support.
-Extend broker::Cluster interface to capture transaction context and completion.
-Running brokers exchange TX information.
-New broker update includes TX information.
-
-** TODO [#B] DTX transaction support.
-Extend broker::Cluster interface to capture transaction context and completion.
-Running brokers exchange DTX information.
-New broker update includes DTX information.
-** TODO [#B] Async completion of accept.
-When this is fixed in the standalone broker, it should be fixed for cluster.
+Updating queue/exchange/binding objects is via the same encode/decode
+that is used by the store. Updatee to use recovery interfaces to
+recover?
+
+** TODO [#A] Failover updates to client.
+Implement the amq.failover exchange to notify clients of membership.
+
+** TODO [#B] Initial status protocol.
+Handshake to give status of each broker member to new members joining.
+Status includes
+- persistent store state (clean, dirty)
+- cluster protocol version.
-** TODO [#B] Persistence support.
-InitialStatus protoocl etc. to support persistent start-up (existing code)
+** TODO [#B] Persistent cluster support.
+Initial status protoocl to support persistent start-up (see existing code)
Only one broker recovers from store, update to others.
Assign cluster IDs to messages recovered from store, don't replicate. See Queue::recover.
-** TODO [#B] Handle other ways that messages can leave a queue.
-
-Ways other than a consumer that messages are taken off a queue.
-
+** TODO [#B] Management support
+Replicate management methods that modify queues - e.g. move, purge.
Target broker may not have all messages on other brokers for purge/destroy.
- Queue::move() - need to wait for lock? Replicate?
- Queue::get() - ???
@@ -171,6 +169,26 @@ Target broker may not have all messages
Need to add callpoints & mcast messages to replicate these?
+** TODO [#B] TX transaction support.
+Extend broker::Cluster interface to capture transaction context and completion.
+Running brokers exchange TX information.
+New broker update includes TX information.
+
+ // FIXME aconway 2010-10-18: As things stand the cluster is not
+ // compatible with transactions
+ // - enqueues occur after routing is complete
+ // - no call to Cluster::enqueue, should be in Queue::process?
+ // - no transaction context associated with messages in the Cluster interface.
+ // - no call to Cluster::accept in Queue::dequeueCommitted
+
+** TODO [#B] DTX transaction support.
+Extend broker::Cluster interface to capture transaction context and completion.
+Running brokers exchange DTX information.
+New broker update includes DTX information.
+
+** TODO [#B] Async completion of accept.
+When this is fixed in the standalone broker, it should be fixed for cluster.
+
** TODO [#B] Network partitions and quorum.
Re-use existing implementation.
@@ -209,25 +227,27 @@ The old cluster has workarounds in the b
- [ ] drop security workarounds: cluster code now operates after message decoding.
- [ ] drop connection tracking in cluster code.
- [ ] simper inconsistent-error handling code, no need to stall.
-** TODO [#C] Support for live updates.
+** TODO [#C] Support for live upgrades.
+
Allow brokers in a running cluster to be replaced one-by-one with a new version.
The old cluster protocol was unstable because any changes in broker
state caused changes to the cluster protocol.The new design should be
much more stable.
-TODO: think about strategies for allowing live updates while extending
-the cluster protocol
-
-
-
-
+Points to implement:
+- Brokers should ignore unknown controls (with a warning) rather than an error.
+- Limit logging frequency for unknown control warnings.
+- Add a version number at front of every CPG message. Determines how the
+ rest of the message is decoded. (allows for entirely new encodings e.g. AMQP 1.0)
+- Protocol version XML element in cluster.xml, on each control.
+- Initial status protocol to include protocol version number.
** TODO [#C] Support for AMQP 1.0.
* Testing
** TODO [#A] Pass all existing cluster tests.
-Requires [[Defer and async complete wiring commands.]]
+Requires [[Defer and async completion of wiring commands.]]
** TODO [#A] New cluster tests.
Stress tests & performance benchmarks focused on changes in new cluster:
- concurrency by queues rather than connections.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:commits-subscribe@qpid.apache.org