You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@qpid.apache.org by Gordon Sim <gs...@redhat.com> on 2012/07/09 14:31:23 UTC

Review Request: qpidd refactor

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

Review request for qpid, Alan Conway and Kenneth Giusti.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
already a little stale (patch was generated against r1331342) and
needs a rebase. However I think the basic ideas involved are clear
enough that it would be worth getting some early feedback.

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* rebase again to latest trunk

* update cmake build (and test on windows)

* fix xml exchange

* c++ unit test cleanup (remove or re-enable those tests that have
  been commented out)

* test against store(s), consider how this might merge with store
  interface redesign

* rethink/clarify role of MessageDistributor v. Messages?

* can QueueCursors be made more like std iterators?


Diffs
-----

  /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
  /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
  /trunk/qpid/cpp/src/Makefile.am 1331342 
  /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
  /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
  /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
  /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
  /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
  /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
  /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
  /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
  /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
  /trunk/qpid/cpp/src/replication.mk 1331342 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
  /trunk/qpid/specs/management-schema.xml 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 11, 2012, 7:28 p.m., Kenneth Giusti wrote:
> > /trunk/qpid/cpp/src/qpid/broker/Queue.cpp, line 269
> > <https://reviews.apache.org/r/5833/diff/1/?file=120493#file120493line269>
> >
> >     Why pass msg by copy instead of reference?

The intention was to enforce the fact that each 'queue' can only operate on a copy, and not affect the message as passed along to other 'queues'. I could also make it a const reference, which might make more sense as the queue implementation can then copy at the point it wants. (That is currently complicated by the fact that we may want to modify the message before we write it to disk, which happens before we add it to Messages, which is where the 'canonical' copy of the message on this queue will be held. However, copying is at present a relatively cheap thing unless there are very many annotations).

I'll see if I can change to a const reference easily. Good comment!


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9080
-----------------------------------------------------------


On July 9, 2012, 12:31 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 9, 2012, 12:31 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
>   /trunk/qpid/cpp/src/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/replication.mk 1331342 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
>   /trunk/qpid/specs/management-schema.xml 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 11, 2012, 7:28 p.m., Kenneth Giusti wrote:
> > /trunk/qpid/cpp/src/qpid/broker/Queue.cpp, line 269
> > <https://reviews.apache.org/r/5833/diff/1/?file=120493#file120493line269>
> >
> >     Why pass msg by copy instead of reference?
> 
> Gordon Sim wrote:
>     The intention was to enforce the fact that each 'queue' can only operate on a copy, and not affect the message as passed along to other 'queues'. I could also make it a const reference, which might make more sense as the queue implementation can then copy at the point it wants. (That is currently complicated by the fact that we may want to modify the message before we write it to disk, which happens before we add it to Messages, which is where the 'canonical' copy of the message on this queue will be held. However, copying is at present a relatively cheap thing unless there are very many annotations).
>     
>     I'll see if I can change to a const reference easily. Good comment!

I did look at doing this, but it required taking a copy in the Queue::deliver() method anyway, as at present enqueue may modify the message (before storing it) and the modified message should then be passed tp push(). Seems in fact simpler/clearer/cleaner at present to leave the deliver() method as taking a copy in my view.


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9080
-----------------------------------------------------------


On July 17, 2012, 12:38 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 12:38 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Kenneth Giusti <kg...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9080
-----------------------------------------------------------

Ship it!


Ship It!


/trunk/qpid/cpp/src/qpid/broker/Queue.cpp
<https://reviews.apache.org/r/5833/#comment19230>

    Why pass msg by copy instead of reference?



/trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp
<https://reviews.apache.org/r/5833/#comment19251>

    +1


- Kenneth Giusti


On July 9, 2012, 12:31 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 9, 2012, 12:31 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
>   /trunk/qpid/cpp/src/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/replication.mk 1331342 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
>   /trunk/qpid/specs/management-schema.xml 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Andrew Stitcher <as...@apache.org>.


> On July 9, 2012, 6:08 p.m., Andrew Stitcher wrote:
> > I haven't looked at this change in detail, what I've looked at seems good.
> > 
> > One comment though, I'd strongly recommend breaking this change up into smaller changesets - This will make it much easier to review I think.
> > 
> > For example I can see that one (seemingly) separate change here is changing the interface to Codec::encode. This change looks (to me) to be self evidently what the interface should be and so should be applied separately - I'd say it's simple enough just to make that change without further review. There may well be other similar changes here.
> >
> 
> Gordon Sim wrote:
>     Yes, the Codec change should not be in this patch. The cluster changes are also irrelevant. I'll have a scan through and get rid of these and others for the next patch. However I think the reality is that this will be a very large patch, simply due to the ripple effect of the central change to the way messages are encapsulated. (E.g. try deleting QueuedMessage and editing Message to not be RefCounted and see the impact of simply getting that to compile, let alone run and pass all the tests!)

It certainly will be large, so taking extraneous changes from consideration will make it easier to review that's all.


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review8977
-----------------------------------------------------------


On July 9, 2012, 12:31 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 9, 2012, 12:31 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
>   /trunk/qpid/cpp/src/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/replication.mk 1331342 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
>   /trunk/qpid/specs/management-schema.xml 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 9, 2012, 6:08 p.m., Andrew Stitcher wrote:
> > I haven't looked at this change in detail, what I've looked at seems good.
> > 
> > One comment though, I'd strongly recommend breaking this change up into smaller changesets - This will make it much easier to review I think.
> > 
> > For example I can see that one (seemingly) separate change here is changing the interface to Codec::encode. This change looks (to me) to be self evidently what the interface should be and so should be applied separately - I'd say it's simple enough just to make that change without further review. There may well be other similar changes here.
> >

Yes, the Codec change should not be in this patch. The cluster changes are also irrelevant. I'll have a scan through and get rid of these and others for the next patch. However I think the reality is that this will be a very large patch, simply due to the ripple effect of the central change to the way messages are encapsulated. (E.g. try deleting QueuedMessage and editing Message to not be RefCounted and see the impact of simply getting that to compile, let alone run and pass all the tests!)


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review8977
-----------------------------------------------------------


On July 9, 2012, 12:31 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 9, 2012, 12:31 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
>   /trunk/qpid/cpp/src/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/replication.mk 1331342 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
>   /trunk/qpid/specs/management-schema.xml 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Andrew Stitcher <as...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review8977
-----------------------------------------------------------


I haven't looked at this change in detail, what I've looked at seems good.

One comment though, I'd strongly recommend breaking this change up into smaller changesets - This will make it much easier to review I think.

For example I can see that one (seemingly) separate change here is changing the interface to Codec::encode. This change looks (to me) to be self evidently what the interface should be and so should be applied separately - I'd say it's simple enough just to make that change without further review. There may well be other similar changes here.


- Andrew Stitcher


On July 9, 2012, 12:31 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 9, 2012, 12:31 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/examples/messaging/spout.cpp 1331342 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1331342 
>   /trunk/qpid/cpp/src/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/amqp_0_10/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SecureConnection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.h 1331342 
>   /trunk/qpid/cpp/src/qpid/client/TCPConnector.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Cluster.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/Connection.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ConnectionCodec.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/ExpiryPolicy.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/FailoverExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateClient.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateDataExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/cluster/UpdateExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/framing/ProtocolInitiation.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1331342 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/MessageImpl.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/ReceiverImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/messaging/SenderImpl.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/AsynchIOHandler.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/Codec.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.h 1331342 
>   /trunk/qpid/cpp/src/qpid/sys/cyrus/CyrusSecurityLayer.cpp 1331342 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1331342 
>   /trunk/qpid/cpp/src/replication.mk 1331342 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1331342 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1331342 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1331342 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1331342 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1331342 
>   /trunk/qpid/specs/management-schema.xml 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1331342 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1331342 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 30, 2012, 4 p.m., Kenneth Giusti wrote:
> > /trunk/qpid/cpp/src/qpid/broker/MessageMap.h, line 9
> > <https://reviews.apache.org/r/5833/diff/1-4/?file=120484#file120484line9>
> >
> >     typo

Thanks! How did I miss that...


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9593
-----------------------------------------------------------


On July 24, 2012, 11:13 a.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 24, 2012, 11:13 a.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> a little stale again (patch was generated against r13613342) and
> needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * test on windows
> 
> * test against store(s)
> 
> * find and fix bug causing intermittent failure of ha failover test
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageUtils.h 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me apart from one ha test which fails frequently (but not always).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Kenneth Giusti <kg...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9593
-----------------------------------------------------------

Ship it!


Ship It!


/trunk/qpid/cpp/src/qpid/broker/MessageMap.h
<https://reviews.apache.org/r/5833/#comment20461>

    typo


- Kenneth Giusti


On July 24, 2012, 11:13 a.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 24, 2012, 11:13 a.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> a little stale again (patch was generated against r13613342) and
> needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * test on windows
> 
> * test against store(s)
> 
> * find and fix bug causing intermittent failure of ha failover test
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageUtils.h 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me apart from one ha test which fails frequently (but not always).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 31, 2012, 9:33 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp, line 268
> > <https://reviews.apache.org/r/5833/diff/5/?file=131497#file131497line268>
> >
> >     Comment typo: qm.position should be m.getSequeunce()

Thanks! Fixed.


> On July 31, 2012, 9:33 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp, line 71
> > <https://reviews.apache.org/r/5833/diff/5/?file=131493#file131493line71>
> >
> >     The completion isn't really part of the persistence context. Is there a more generic message context they could be part of?

I agree its slightly misplaced. It is where it is because the store requires access to it through the PersistableMessage interface and I was keen to avoid the changes rippling too far. However, that doesn't mean that's the only interface through which it can be obtained. I have massaged this a little to (I hope) fix the concern.


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9667
-----------------------------------------------------------


On July 31, 2012, 8:18 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 31, 2012, 8:18 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> a little stale again (patch was generated against r13613342) and
> needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * test on windows
> 
> 
> This addresses bug QPID-4178.
>     https://issues.apache.org/jira/browse/QPID-4178
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1367554 
>   /trunk/qpid/cpp/src/Makefile.am 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1367554 
>   /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/replication.mk 1367554 
>   /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367554 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1367554 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/MessageUtils.h 1367554 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1367554 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1367554 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1367554 
>   /trunk/qpid/specs/management-schema.xml 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367554 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes.
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9667
-----------------------------------------------------------


I re-reviewed the ha changes, all looks good.


/trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp
<https://reviews.apache.org/r/5833/#comment20593>

    The completion isn't really part of the persistence context. Is there a more generic message context they could be part of?



/trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp
<https://reviews.apache.org/r/5833/#comment20599>

    Comment typo: qm.position should be m.getSequeunce()


- Alan Conway


On July 31, 2012, 8:18 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 31, 2012, 8:18 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> a little stale again (patch was generated against r13613342) and
> needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * test on windows
> 
> 
> This addresses bug QPID-4178.
>     https://issues.apache.org/jira/browse/QPID-4178
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1367554 
>   /trunk/qpid/cpp/src/Makefile.am 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1367554 
>   /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367554 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367554 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367554 
>   /trunk/qpid/cpp/src/replication.mk 1367554 
>   /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367554 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1367554 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/MessageUtils.h 1367554 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1367554 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367554 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1367554 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1367554 
>   /trunk/qpid/specs/management-schema.xml 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367554 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367554 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes.
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Kenneth Giusti <kg...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9826
-----------------------------------------------------------

Ship it!


Acking revs 5 thru 10.

- Kenneth Giusti


On Aug. 2, 2012, 9:17 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated Aug. 2, 2012, 9:17 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> a little stale again (patch was generated against r13613342) and
> needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * test on windows
> 
> 
> This addresses bug QPID-4178.
>     https://issues.apache.org/jira/browse/QPID-4178
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
>   /trunk/qpid/cpp/src/Makefile.am 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
>   /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
>   /trunk/qpid/cpp/src/replication.mk 1367797 
>   /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes.
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated Aug. 2, 2012, 9:17 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

One final missing export declaration for windows.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/Makefile.am 1367797 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
  /trunk/qpid/cpp/src/replication.mk 1367797 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated Aug. 2, 2012, 8:25 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Further tweaks for windows build.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/Makefile.am 1367797 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
  /trunk/qpid/cpp/src/replication.mk 1367797 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated Aug. 2, 2012, 7:19 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Fixes for windows build


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/Makefile.am 1367797 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
  /trunk/qpid/cpp/src/replication.mk 1367797 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated Aug. 1, 2012, 7:47 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Minor store related fixes in Queue.cpp exposed by durable federation tests.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/Makefile.am 1367797 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
  /trunk/qpid/cpp/src/replication.mk 1367797 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated Aug. 1, 2012, 12:38 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Minor adjustments per Alan's feedback.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/Makefile.am 1367797 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367797 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367797 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367797 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367797 
  /trunk/qpid/cpp/src/replication.mk 1367797 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367797 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367797 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367797 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367797 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367797 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367797 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367797 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367797 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 31, 2012, 8:18 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


This addresses bug QPID-4178.
    https://issues.apache.org/jira/browse/QPID-4178


Diffs
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367554 
  /trunk/qpid/cpp/src/Makefile.am 1367554 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367554 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367554 
  /trunk/qpid/cpp/src/replication.mk 1367554 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367554 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367554 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367554 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367554 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367554 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367554 
  /trunk/qpid/specs/management-schema.xml 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367554 

Diff: https://reviews.apache.org/r/5833/diff/


Testing (updated)
-------

With clustering disabled, make check passes.


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 31, 2012, 8:07 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

HA test failure fixed; tested against linux based store plugin (see separate patch for some changes there) and those tests pass now also; minor cleanup (e.g. Ken's last point).


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1367554 
  /trunk/qpid/cpp/src/Makefile.am 1367554 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1367554 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1367554 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1367554 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1367554 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1367554 
  /trunk/qpid/cpp/src/replication.mk 1367554 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1367554 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/Makefile.am 1367554 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1367554 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1367554 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1367554 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1367554 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1367554 
  /trunk/qpid/specs/management-schema.xml 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1367554 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1367554 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me apart from one ha test which fails frequently (but not always).


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 31, 2012, 8:06 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Description (updated)
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* test on windows


Diffs
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/Makefile.am 1362014 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
  /trunk/qpid/cpp/src/replication.mk 1362014 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1362014 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
  /trunk/qpid/specs/management-schema.xml 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me apart from one ha test which fails frequently (but not always).


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 24, 2012, 11:13 a.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Brought main description up to date.


Description (updated)
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
a little stale again (patch was generated against r13613342) and
needs (yet) another rebase. However it is getting to the point where I'll be asking to commit it soon, so if anyone has feedback, now is the time to give it!

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* rebase again to latest trunk

* test on windows

* test against store(s)

* find and fix bug causing intermittent failure of ha failover test


Diffs
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/Makefile.am 1362014 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
  /trunk/qpid/cpp/src/replication.mk 1362014 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1362014 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
  /trunk/qpid/specs/management-schema.xml 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 

Diff: https://reviews.apache.org/r/5833/diff/


Testing (updated)
-------

With clustering disabled, make check passes for me apart from one ha test which fails frequently (but not always).


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 24, 2012, 9 a.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Fixed CMake build; cleaned up and re-enabled unit tests; Altered Queue::seek() signature following Alan's comments (went with overloaded methods as opposed to a boost optional).


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
already a little stale (patch was generated against r1331342) and
needs a rebase. However I think the basic ideas involved are clear
enough that it would be worth getting some early feedback.

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* rebase again to latest trunk

* update cmake build (and test on windows)

* fix xml exchange

* c++ unit test cleanup (remove or re-enable those tests that have
  been commented out)

* test against store(s), consider how this might merge with store
  interface redesign

* rethink/clarify role of MessageDistributor v. Messages?

* can QueueCursors be made more like std iterators?


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/Makefile.am 1362014 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
  /trunk/qpid/cpp/src/qpid/store/MessageStorePlugin.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
  /trunk/qpid/cpp/src/replication.mk 1362014 
  /trunk/qpid/cpp/src/tests/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageUtils.h 1362014 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
  /trunk/qpid/specs/management-schema.xml 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).


Thanks,

Gordon Sim

RE: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.

On Thu, 2012-07-19 at 09:06 -0400, Nitin Shah wrote:
> We are using 0-16 (c++ brokers etc..) release. Does this affect backward compatibility? We might not want to move to the next release unless there is no impact on our implementations. Can someone comment and put me straight.
> 

This change is not going to be part of the next release - 0.18 currently
in beta.

> Thanks
> Nitin
> 
> 
> 
> -----Original Message-----
> From: Alan Conway [mailto:noreply@reviews.apache.org] On Behalf Of Alan Conway
> Sent: Wednesday, July 18, 2012 3:10 PM
> To: Kenneth Giusti; Alan Conway
> Cc: Gordon Sim; qpid
> Subject: Re: Review Request: qpidd refactor
> 
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/#review9252
> -----------------------------------------------------------
> 
> Ship it!
> 
> 
> Ship It!
> 
> - Alan Conway
> 
> 
> On July 17, 2012, 7:46 p.m., Gordon Sim wrote:
> >
> > -----------------------------------------------------------
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/5833/
> > -----------------------------------------------------------
> >
> > (Updated July 17, 2012, 7:46 p.m.)
> >
> >
> > Review request for qpid, Alan Conway and Kenneth Giusti.
> >
> >
> > Description
> > -------
> >
> > == Background ==
> >
> > I've been looking at what would be required to get AMQP 1.0 support in
> > the qpidd broker (using proton-c). In that context I felt there was a
> > need to refactor the broker code, particularly that part that would be
> > shared between different protocol versions. Part of the motivation was
> > clearer separation of 0-10 specific logic, so that 1.0 logic could be
> > introduced as an alternative. However part of it was also simply the
> > recognition of some long-standing problems that we have never stopped
> > to address.
> >
> > So, here is a patch representing my ideas on what is needed. This is
> > already a little stale (patch was generated against r1331342) and
> > needs a rebase. However I think the basic ideas involved are clear
> > enough that it would be worth getting some early feedback.
> >
> > == Key Changes ==
> >
> > qpid::broker::Message
> >
> > This is now supposed to be a protocol neutral representation of a
> > message. It no longer exposes qpid::framing::FrameSet. It can be based
> > on data received in different encodings (this patch only includes the
> > existing 0-10 encoding).
> >
> > The immutable, sharable state is separated from the mutable
> > queue-specific state. Messages themselves are no longer held through a
> > shared pointer but are passed by reference or copied if needed. The
> > immutable state (essentially the data as received) *is* still shared
> > and referenced internally through an intrusive pointer. There is no
> > longer a message level lock. A message instance is 'owned' by
> > someother entity (usually the queue it is on) which controls
> > concurrent access/modification if necessary.
> >
> > The persistence context is a separate part of the message also.
> > Currently that can be shared between two message instances if desired.
> >
> > qpid::broker::Messages
> >
> > Switched from using qpid::broker::QueuedMessage (which relied on
> > shared pointer to message itself and made sequence number the explicit
> > - and only - way to refer to a specific message) to using modified
> > Message class directly and a new qpid::broker::QueueCursor.
> >
> > The cursor is opaque outside the Messages implementation to which it
> > relates. It provides a way to refer to a specific message (without
> > directly using sequence number, though at present that is what is used
> > 'under the covers') and/or to track progress through a sequence of
> > messages (for consumers or other iterating entities).
> >
> > I.e. its an iterator to a Message within its containing Messages
> > instance that is not invalidated by changes to that container.
> >
> > A Messages instance *owns* the Message instances within it. Other
> > classes access this through a reference or (raw) pointer, or if needed
> > copy it (the immutable part can be - and is - safely shared).
> >
> > The codepath for browse/consume is a lot more unified now. You use a
> > cursor and call Messages::next() in each case. This also lays the
> > foundation for selectors.
> >
> > The simplified Messages interface led to a simplied
> > MessageDistributor. There is still a little more to do to clarify
> > these separate roles (or indeed perhaps unify them?) but more on that
> > later.
> >
> > qpid::broker::amqp_0_10::MessageTransfer
> >
> > This represents the familiar 0-10 encoding of a message. This class is
> > broadly similar to the old Message class, based on a FrameSet. However
> > it represents the shared and essentially immutable state. The
> > sendHeader() method now explicitly takes a copy of the original
> > headers and adds to it or otherwise modifies it if needed (e.g. for
> > redelivered flag, ttl, annotations etc).
> >
> > [Ideally I'd like to move more of the 0-10 specific classes out of
> > qpid::broker and into qpid::broker::amqp_0_10, but that has no
> > functional relevance so I've left existing classes alone for now.]
> >
> > qpid::broker::Consumer
> >
> > The deliver() method now takes a QueueCursor (representing a 'handle'
> > to this message for use in subsequent operations such as accept,
> > relese etc) and a *constant reference* to the Message itself (i.e.
> > consumers can't alter the state of the message on the queue directly,
> > but only through operations on the queue itself).
> >
> > qpid::broker::QueueRegistry
> >
> > The actual queue creation has been pulled out into a base class,
> > QueueFactory. The actual class of the Queue returned can now be varied
> > and there are two subclasses in the current patch. The first is a
> > replacement for the ring policy logic, whereby messages are removed
> > from the queue in order to keep the queue from growing above a
> > configured limit. The second is for last value queues and simply pulls
> > the special case behaviour out of the common code path.
> >
> > The handling of queue configuration has also been made cleaner and
> > more uniform, based on the QueueSettings class.
> >
> > qpid::broker::QueuePolicy
> >
> > This class has been removed. There is a new QueueDepth utility used
> > for configuring limits, tracking current depth and testing the latter
> > against the former. This is used directly by Queue. The behaviour at
> > the limit can be varied by subclassing queue.
> >
> > == Limitations etc ==
> >
> > clustering
> >
> > This breaks clustering. Indeed it will not compile unless clustering
> > is disabled (--without-cpg in configure). Keeping the cluster code in
> > sync was distracting me from the core goal, given its entanglement
> > with the broker code.
> >
> > My assumption is that the new ha code will eventually replace the
> > cluster anyway and the amount of change that would be required to get
> > the cluster working with this refactor may not be worth it and may in
> > fact undermine its stability anyway (which seem the only good argument
> > for using it).
> >
> > I don't believe there is anything insurmountable to do to re-enable
> > cluster if that was desired however.
> >
> > old & nasty features removed
> >
> > I have removed support for flow to disk, the legacy version of lvq
> > with two modes (the updated version of lvq is of course still
> > functional), the last-man-standing persistence in clustering and the
> > old async queue replication. They are really quite horribly
> > implemented and/or are no longer necessary in my view.
> >
> > == Still To Do ==
> >
> > * rebase again to latest trunk
> >
> > * update cmake build (and test on windows)
> >
> > * fix xml exchange
> >
> > * c++ unit test cleanup (remove or re-enable those tests that have
> >   been commented out)
> >
> > * test against store(s), consider how this might merge with store
> >   interface redesign
> >
> > * rethink/clarify role of MessageDistributor v. Messages?
> >
> > * can QueueCursors be made more like std iterators?
> >
> >
> > Diffs
> > -----
> >
> >   /trunk/qpid/cpp/src/CMakeLists.txt 1362014
> >   /trunk/qpid/cpp/src/Makefile.am 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014
> >   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014
> >   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014
> >   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014
> >   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014
> >   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014
> >   /trunk/qpid/cpp/src/replication.mk 1362014
> >   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/Makefile.am 1362014
> >   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION
> >   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014
> >   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014
> >   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014
> >   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014
> >   /trunk/qpid/specs/management-schema.xml 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014
> >   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014
> >
> > Diff: https://reviews.apache.org/r/5833/diff/
> >
> >
> > Testing
> > -------
> >
> > With clustering disabled, make check passes for me. However I have
> > just realised I've not got the xml exchange enabled either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> >
> >
> > Thanks,
> >
> > Gordon Sim
> >
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

On 07/19/2012 02:06 PM, Nitin Shah wrote:
> We are using 0-16 (c++ brokers etc..) release. Does this affect backward compatibility? We might not want to move to the next release unless there is no impact on our implementations. Can someone comment and put me straight.

I've posted a more complete mail to the user list, detailing the 
features affected. The patch is still very much at the proposal stage 
and as Alan stated will certainly not affect 0.18 even if accepted.

(The 'Ship it!' comment is just the reviewboard shorthand for a reviewer 
to indicate they have no concerns. Its not as alarming as it may sound!)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

RE: Review Request: qpidd refactor

Posted by Nitin Shah <ns...@btisystems.com>.

We are using 0-16 (c++ brokers etc..) release. Does this affect backward compatibility? We might not want to move to the next release unless there is no impact on our implementations. Can someone comment and put me straight.

Thanks
Nitin



-----Original Message-----
From: Alan Conway [mailto:noreply@reviews.apache.org] On Behalf Of Alan Conway
Sent: Wednesday, July 18, 2012 3:10 PM
To: Kenneth Giusti; Alan Conway
Cc: Gordon Sim; qpid
Subject: Re: Review Request: qpidd refactor


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9252
-----------------------------------------------------------

Ship it!


Ship It!

- Alan Conway


On July 17, 2012, 7:46 p.m., Gordon Sim wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
>
> (Updated July 17, 2012, 7:46 p.m.)
>
>
> Review request for qpid, Alan Conway and Kenneth Giusti.
>
>
> Description
> -------
>
> == Background ==
>
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
>
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
>
> == Key Changes ==
>
> qpid::broker::Message
>
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
>
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
>
> The persistence context is a separate part of the message also.
> Currently that can be shared between two message instances if desired.
>
> qpid::broker::Messages
>
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
>
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
>
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
>
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
>
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
>
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
>
> qpid::broker::amqp_0_10::MessageTransfer
>
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
>
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
>
> qpid::broker::Consumer
>
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself (i.e.
> consumers can't alter the state of the message on the queue directly,
> but only through operations on the queue itself).
>
> qpid::broker::QueueRegistry
>
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
>
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
>
> qpid::broker::QueuePolicy
>
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
>
> == Limitations etc ==
>
> clustering
>
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
>
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
>
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
>
> old & nasty features removed
>
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
>
> == Still To Do ==
>
> * rebase again to latest trunk
>
> * update cmake build (and test on windows)
>
> * fix xml exchange
>
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
>
> * test against store(s), consider how this might merge with store
>   interface redesign
>
> * rethink/clarify role of MessageDistributor v. Messages?
>
> * can QueueCursors be made more like std iterators?
>
>
> Diffs
> -----
>
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014
>   /trunk/qpid/cpp/src/Makefile.am 1362014
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014
>   /trunk/qpid/cpp/src/replication.mk 1362014
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014
>   /trunk/qpid/specs/management-schema.xml 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014
>
> Diff: https://reviews.apache.org/r/5833/diff/
>
>
> Testing
> -------
>
> With clustering disabled, make check passes for me. However I have
> just realised I've not got the xml exchange enabled either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
>
>
> Thanks,
>
> Gordon Sim
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9252
-----------------------------------------------------------

Ship it!


Ship It!

- Alan Conway


On July 17, 2012, 7:46 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 7:46 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 17, 2012, 7:46 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Changes based on Alans feedback (didn't yet make any attempt to expand use of boost as suggested).


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
already a little stale (patch was generated against r1331342) and
needs a rebase. However I think the basic ideas involved are clear
enough that it would be worth getting some early feedback.

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* rebase again to latest trunk

* update cmake build (and test on windows)

* fix xml exchange

* c++ unit test cleanup (remove or re-enable those tests that have
  been commented out)

* test against store(s), consider how this might merge with store
  interface redesign

* rethink/clarify role of MessageDistributor v. Messages?

* can QueueCursors be made more like std iterators?


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/Makefile.am 1362014 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
  /trunk/qpid/cpp/src/replication.mk 1362014 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
  /trunk/qpid/specs/management-schema.xml 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).


Thanks,

Gordon Sim

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp, line 35
> > <https://reviews.apache.org/r/5833/diff/2/?file=123565#file123565line35>
> >
> >     const Message&?

Agreed.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp, line 41
> > <https://reviews.apache.org/r/5833/diff/2/?file=123565#file123565line41>
> >
> >     Could say just "return m.getState() ==..."

Agreed.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp, line 74
> > <https://reviews.apache.org/r/5833/diff/2/?file=123609#file123609line74>
> >
> >     Oops! Code inside an assert()! Will explode on in release build.

Oops indeed! Will fix that right away.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp, line 136
> > <https://reviews.apache.org/r/5833/diff/2/?file=123609#file123609line136>
> >
> >     Not important but: why pass SequenceNumber by reference? It's just an int32_t

Agreed.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp, line 57
> > <https://reviews.apache.org/r/5833/diff/2/?file=123613#file123613line57>
> >
> >     Maybe move this down closer to its use, its a bit puzzling when seen alone.

Sure.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp, line 130
> > <https://reviews.apache.org/r/5833/diff/2/?file=123613#file123613line130>
> >
> >     There  is still a bug here (not introduced by you though.) Seek from 0 won't get the front of the queue if the sequence numbers have wrapped. Want to make sure we leave a comment or note of that somewhere.

I think that bug is fixed, as we now no longer do any sequence number comparison if we simply want the front message. I.e. this method no longer just calls getNext() with position 0. (The 0 in the last parameter is a pointer to a SequenceNumber, so 0 means null not actual 0 value).

Thanks for all the excellent feedback!


- Gordon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9213
-----------------------------------------------------------


On July 17, 2012, 12:38 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 12:38 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.


> On July 17, 2012, 3:01 p.m., Alan Conway wrote:
> > /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp, line 130
> > <https://reviews.apache.org/r/5833/diff/2/?file=123613#file123613line130>
> >
> >     There  is still a bug here (not introduced by you though.) Seek from 0 won't get the front of the queue if the sequence numbers have wrapped. Want to make sure we leave a comment or note of that somewhere.
> 
> Gordon Sim wrote:
>     I think that bug is fixed, as we now no longer do any sequence number comparison if we simply want the front message. I.e. this method no longer just calls getNext() with position 0. (The 0 in the last parameter is a pointer to a SequenceNumber, so 0 means null not actual 0 value).
>     
>     Thanks for all the excellent feedback!

Aha! Good. Passing a SequenceNumber by pointer is a bit icky. boost::optional<SequenceNumber> might make the intent more clear but it's just aesthetic


- Alan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9213
-----------------------------------------------------------


On July 17, 2012, 7:46 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 7:46 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/#review9213
-----------------------------------------------------------


Looked at the HA stuff - looks good, improved in several places e.g. avoiding queue scans now you're using a map rather than a SequenceSet and using seek.  One serious issue - code in an assert, the remaining comments you can take or leave, non of them is important.


/trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp
<https://reviews.apache.org/r/5833/#comment19704>

    const Message&?



/trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp
<https://reviews.apache.org/r/5833/#comment19705>

    Could say just "return m.getState() ==..."



/trunk/qpid/cpp/src/qpid/broker/QueueDepth.h
<https://reviews.apache.org/r/5833/#comment19708>

    Could be simplified with boost::operators. You only have to define ==, <, +=, -= and the template generates the rest.



/trunk/qpid/cpp/src/qpid/broker/QueueDepth.h
<https://reviews.apache.org/r/5833/#comment19706>

    Could use boost::optional



/trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp
<https://reviews.apache.org/r/5833/#comment19701>

    Oops! Code inside an assert()! Will explode on in release build.



/trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp
<https://reviews.apache.org/r/5833/#comment19700>

    Not important but: why pass SequenceNumber by reference? It's just an int32_t



/trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp
<https://reviews.apache.org/r/5833/#comment19702>

    Maybe move this down closer to its use, its a bit puzzling when seen alone.



/trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp
<https://reviews.apache.org/r/5833/#comment19703>

    There  is still a bug here (not introduced by you though.) Seek from 0 won't get the front of the queue if the sequence numbers have wrapped. Want to make sure we leave a comment or note of that somewhere.


- Alan Conway


On July 17, 2012, 12:38 p.m., Gordon Sim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 12:38 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Description
> -------
> 
> == Background ==
> 
> I've been looking at what would be required to get AMQP 1.0 support in
> the qpidd broker (using proton-c). In that context I felt there was a
> need to refactor the broker code, particularly that part that would be
> shared between different protocol versions. Part of the motivation was
> clearer separation of 0-10 specific logic, so that 1.0 logic could be
> introduced as an alternative. However part of it was also simply the
> recognition of some long-standing problems that we have never stopped
> to address.
> 
> So, here is a patch representing my ideas on what is needed. This is
> already a little stale (patch was generated against r1331342) and
> needs a rebase. However I think the basic ideas involved are clear
> enough that it would be worth getting some early feedback.
> 
> == Key Changes ==
> 
> qpid::broker::Message
> 
> This is now supposed to be a protocol neutral representation of a
> message. It no longer exposes qpid::framing::FrameSet. It can be based
> on data received in different encodings (this patch only includes the
> existing 0-10 encoding).
> 
> The immutable, sharable state is separated from the mutable
> queue-specific state. Messages themselves are no longer held through a
> shared pointer but are passed by reference or copied if needed. The
> immutable state (essentially the data as received) *is* still shared
> and referenced internally through an intrusive pointer. There is no
> longer a message level lock. A message instance is 'owned' by
> someother entity (usually the queue it is on) which controls
> concurrent access/modification if necessary.
> 
> The persistence context is a separate part of the message
> also. Currently that can be shared between two message instances if
> desired.
> 
> qpid::broker::Messages
> 
> Switched from using qpid::broker::QueuedMessage (which relied on
> shared pointer to message itself and made sequence number the explicit
> - and only - way to refer to a specific message) to using modified
> Message class directly and a new qpid::broker::QueueCursor.
> 
> The cursor is opaque outside the Messages implementation to which it
> relates. It provides a way to refer to a specific message (without
> directly using sequence number, though at present that is what is used
> 'under the covers') and/or to track progress through a sequence of
> messages (for consumers or other iterating entities).
> 
> I.e. its an iterator to a Message within its containing Messages
> instance that is not invalidated by changes to that container.
> 
> A Messages instance *owns* the Message instances within it. Other
> classes access this through a reference or (raw) pointer, or if needed
> copy it (the immutable part can be - and is - safely shared).
> 
> The codepath for browse/consume is a lot more unified now. You use a
> cursor and call Messages::next() in each case. This also lays the
> foundation for selectors.
> 
> The simplified Messages interface led to a simplied
> MessageDistributor. There is still a little more to do to clarify
> these separate roles (or indeed perhaps unify them?) but more on that
> later.
> 
> qpid::broker::amqp_0_10::MessageTransfer
> 
> This represents the familiar 0-10 encoding of a message. This class is
> broadly similar to the old Message class, based on a FrameSet. However
> it represents the shared and essentially immutable state. The
> sendHeader() method now explicitly takes a copy of the original
> headers and adds to it or otherwise modifies it if needed (e.g. for
> redelivered flag, ttl, annotations etc).
> 
> [Ideally I'd like to move more of the 0-10 specific classes out of
> qpid::broker and into qpid::broker::amqp_0_10, but that has no
> functional relevance so I've left existing classes alone for now.]
> 
> qpid::broker::Consumer
> 
> The deliver() method now takes a QueueCursor (representing a 'handle'
> to this message for use in subsequent operations such as accept,
> relese etc) and a *constant reference* to the Message itself
> (i.e. consumers can't alter the state of the message on the queue
> directly, but only through operations on the queue itself).
> 
> qpid::broker::QueueRegistry
> 
> The actual queue creation has been pulled out into a base class,
> QueueFactory. The actual class of the Queue returned can now be varied
> and there are two subclasses in the current patch. The first is a
> replacement for the ring policy logic, whereby messages are removed
> from the queue in order to keep the queue from growing above a
> configured limit. The second is for last value queues and simply pulls
> the special case behaviour out of the common code path.
> 
> The handling of queue configuration has also been made cleaner and
> more uniform, based on the QueueSettings class.
> 
> qpid::broker::QueuePolicy
> 
> This class has been removed. There is a new QueueDepth utility used
> for configuring limits, tracking current depth and testing the latter
> against the former. This is used directly by Queue. The behaviour at
> the limit can be varied by subclassing queue.
> 
> == Limitations etc ==
> 
> clustering
> 
> This breaks clustering. Indeed it will not compile unless clustering
> is disabled (--without-cpg in configure). Keeping the cluster code in
> sync was distracting me from the core goal, given its entanglement
> with the broker code.
> 
> My assumption is that the new ha code will eventually replace the
> cluster anyway and the amount of change that would be required to get
> the cluster working with this refactor may not be worth it and may in
> fact undermine its stability anyway (which seem the only good argument
> for using it).
> 
> I don't believe there is anything insurmountable to do to re-enable
> cluster if that was desired however.
> 
> old & nasty features removed
> 
> I have removed support for flow to disk, the legacy version of lvq
> with two modes (the updated version of lvq is of course still
> functional), the last-man-standing persistence in clustering and the
> old async queue replication. They are really quite horribly
> implemented and/or are no longer necessary in my view.
> 
> == Still To Do ==
> 
> * rebase again to latest trunk
> 
> * update cmake build (and test on windows)
> 
> * fix xml exchange
> 
> * c++ unit test cleanup (remove or re-enable those tests that have
>   been commented out)
> 
> * test against store(s), consider how this might merge with store
>   interface redesign
> 
> * rethink/clarify role of MessageDistributor v. Messages?
> 
> * can QueueCursors be made more like std iterators?
> 
> 
> Diffs
> -----
> 
>   /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
>   /trunk/qpid/cpp/src/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
>   /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
>   /trunk/qpid/cpp/src/replication.mk 1362014 
>   /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
>   /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
>   /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
>   /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
>   /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
>   /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
>   /trunk/qpid/specs/management-schema.xml 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
>   /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 
> 
> Diff: https://reviews.apache.org/r/5833/diff/
> 
> 
> Testing
> -------
> 
> With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
> either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).
> 
> 
> Thanks,
> 
> Gordon Sim
> 
>

Re: Review Request: qpidd refactor

Posted by Alan Conway <ac...@redhat.com>.

On Tue, 2012-07-17 at 12:38 +0000, Gordon Sim wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5833/
> -----------------------------------------------------------
> 
> (Updated July 17, 2012, 12:38 p.m.)
> 
> 
> Review request for qpid, Alan Conway and Kenneth Giusti.
> 
> 
> Changes
> -------
> 
> Rebased (to r1362014). Removed irrelevant changes (to cluster and to Codec interface). Fixed xml exchange.
> 
> Current status: there is one intermittent failure in the ha tests (ha_tests.LongTests.test_failover_send_receive); some unit tests still disabled; all other tests are passing reliably.

That test is failing intermittently on trunk/0.18 also so that may not
be related to your changes.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Review Request: qpidd refactor

Posted by Gordon Sim <gs...@redhat.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5833/
-----------------------------------------------------------

(Updated July 17, 2012, 12:38 p.m.)


Review request for qpid, Alan Conway and Kenneth Giusti.


Changes
-------

Rebased (to r1362014). Removed irrelevant changes (to cluster and to Codec interface). Fixed xml exchange.

Current status: there is one intermittent failure in the ha tests (ha_tests.LongTests.test_failover_send_receive); some unit tests still disabled; all other tests are passing reliably.


Description
-------

== Background ==

I've been looking at what would be required to get AMQP 1.0 support in
the qpidd broker (using proton-c). In that context I felt there was a
need to refactor the broker code, particularly that part that would be
shared between different protocol versions. Part of the motivation was
clearer separation of 0-10 specific logic, so that 1.0 logic could be
introduced as an alternative. However part of it was also simply the
recognition of some long-standing problems that we have never stopped
to address.

So, here is a patch representing my ideas on what is needed. This is
already a little stale (patch was generated against r1331342) and
needs a rebase. However I think the basic ideas involved are clear
enough that it would be worth getting some early feedback.

== Key Changes ==

qpid::broker::Message

This is now supposed to be a protocol neutral representation of a
message. It no longer exposes qpid::framing::FrameSet. It can be based
on data received in different encodings (this patch only includes the
existing 0-10 encoding).

The immutable, sharable state is separated from the mutable
queue-specific state. Messages themselves are no longer held through a
shared pointer but are passed by reference or copied if needed. The
immutable state (essentially the data as received) *is* still shared
and referenced internally through an intrusive pointer. There is no
longer a message level lock. A message instance is 'owned' by
someother entity (usually the queue it is on) which controls
concurrent access/modification if necessary.

The persistence context is a separate part of the message
also. Currently that can be shared between two message instances if
desired.

qpid::broker::Messages

Switched from using qpid::broker::QueuedMessage (which relied on
shared pointer to message itself and made sequence number the explicit
- and only - way to refer to a specific message) to using modified
Message class directly and a new qpid::broker::QueueCursor.

The cursor is opaque outside the Messages implementation to which it
relates. It provides a way to refer to a specific message (without
directly using sequence number, though at present that is what is used
'under the covers') and/or to track progress through a sequence of
messages (for consumers or other iterating entities).

I.e. its an iterator to a Message within its containing Messages
instance that is not invalidated by changes to that container.

A Messages instance *owns* the Message instances within it. Other
classes access this through a reference or (raw) pointer, or if needed
copy it (the immutable part can be - and is - safely shared).

The codepath for browse/consume is a lot more unified now. You use a
cursor and call Messages::next() in each case. This also lays the
foundation for selectors.

The simplified Messages interface led to a simplied
MessageDistributor. There is still a little more to do to clarify
these separate roles (or indeed perhaps unify them?) but more on that
later.

qpid::broker::amqp_0_10::MessageTransfer

This represents the familiar 0-10 encoding of a message. This class is
broadly similar to the old Message class, based on a FrameSet. However
it represents the shared and essentially immutable state. The
sendHeader() method now explicitly takes a copy of the original
headers and adds to it or otherwise modifies it if needed (e.g. for
redelivered flag, ttl, annotations etc).

[Ideally I'd like to move more of the 0-10 specific classes out of
qpid::broker and into qpid::broker::amqp_0_10, but that has no
functional relevance so I've left existing classes alone for now.]

qpid::broker::Consumer

The deliver() method now takes a QueueCursor (representing a 'handle'
to this message for use in subsequent operations such as accept,
relese etc) and a *constant reference* to the Message itself
(i.e. consumers can't alter the state of the message on the queue
directly, but only through operations on the queue itself).

qpid::broker::QueueRegistry

The actual queue creation has been pulled out into a base class,
QueueFactory. The actual class of the Queue returned can now be varied
and there are two subclasses in the current patch. The first is a
replacement for the ring policy logic, whereby messages are removed
from the queue in order to keep the queue from growing above a
configured limit. The second is for last value queues and simply pulls
the special case behaviour out of the common code path.

The handling of queue configuration has also been made cleaner and
more uniform, based on the QueueSettings class.

qpid::broker::QueuePolicy

This class has been removed. There is a new QueueDepth utility used
for configuring limits, tracking current depth and testing the latter
against the former. This is used directly by Queue. The behaviour at
the limit can be varied by subclassing queue.

== Limitations etc ==

clustering

This breaks clustering. Indeed it will not compile unless clustering
is disabled (--without-cpg in configure). Keeping the cluster code in
sync was distracting me from the core goal, given its entanglement
with the broker code.

My assumption is that the new ha code will eventually replace the
cluster anyway and the amount of change that would be required to get
the cluster working with this refactor may not be worth it and may in
fact undermine its stability anyway (which seem the only good argument
for using it).

I don't believe there is anything insurmountable to do to re-enable
cluster if that was desired however.

old & nasty features removed

I have removed support for flow to disk, the legacy version of lvq
with two modes (the updated version of lvq is of course still
functional), the last-man-standing persistence in clustering and the
old async queue replication. They are really quite horribly
implemented and/or are no longer necessary in my view.

== Still To Do ==

* rebase again to latest trunk

* update cmake build (and test on windows)

* fix xml exchange

* c++ unit test cleanup (remove or re-enable those tests that have
  been commented out)

* test against store(s), consider how this might merge with store
  interface redesign

* rethink/clarify role of MessageDistributor v. Messages?

* can QueueCursors be made more like std iterators?


Diffs (updated)
-----

  /trunk/qpid/cpp/src/CMakeLists.txt 1362014 
  /trunk/qpid/cpp/src/Makefile.am 1362014 
  /trunk/qpid/cpp/src/qpid/broker/AsyncCompletion.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Broker.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Consumer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Deliverable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliverableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryAdapter.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DeliveryRecord.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/DtxAck.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Exchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ExpiryPolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Fairshare.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/FifoDistributor.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/HeadersExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/IndexedDeque.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LegacyLVQ.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Link.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/LossyQueue.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Lvq.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/MapHandler.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/Message.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Message.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageBuilder.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDeque.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageDistributor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageGroupManager.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/MessageMap.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Messages.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Persistable.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PersistableMessage.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/PriorityQueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/Queue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueCursor.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFactory.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueFlowLimit.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueObserver.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueuePolicy.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueRegistry.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueueSettings.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/QueuedMessage.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/RecoveryManagerImpl.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionAdapter.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionHandler.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/ThresholdAlerts.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxAccept.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxBuffer.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOp.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxOpVisitor.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.h 1362014 
  /trunk/qpid/cpp/src/qpid/broker/TxPublish.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.h PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/broker/amqp_0_10/MessageTransfer.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/qpid/ha/Backup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/BrokerReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/Primary.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueGuard.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/QueueReplicator.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/RemoteBackup.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.h 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicatingSubscription.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/ha/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementDirectExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/management/ManagementTopicExchange.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicatingEventListener.cpp 1362014 
  /trunk/qpid/cpp/src/qpid/replication/ReplicationExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/replication/constants.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.h 1362014 
  /trunk/qpid/cpp/src/qpid/xml/XmlExchange.cpp 1362014 
  /trunk/qpid/cpp/src/replication.mk 1362014 
  /trunk/qpid/cpp/src/tests/ClientSessionTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/DeliveryRecordTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ExchangeTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/Makefile.am 1362014 
  /trunk/qpid/cpp/src/tests/MessageBuilderTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/MessageTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueDepth.cpp PRE-CREATION 
  /trunk/qpid/cpp/src/tests/QueueEvents.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueFlowLimitTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueuePolicyTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueRegistryTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/QueueTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ReplicationTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/TxMocks.h 1362014 
  /trunk/qpid/cpp/src/tests/TxPublishTest.cpp 1362014 
  /trunk/qpid/cpp/src/tests/ha_tests.py 1362014 
  /trunk/qpid/cpp/src/tests/test_store.cpp 1362014 
  /trunk/qpid/specs/management-schema.xml 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/alternate_exchange.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/management.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/msg_groups.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/new_api.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/priority.py 1362014 
  /trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py 1362014 

Diff: https://reviews.apache.org/r/5833/diff/


Testing
-------

With clustering disabled, make check passes for me. However I have just realised I've not got the xml exchange enabled
either - though that was inadvertant - and there is some work needed there (similar to what was done in the HeadersExchange).


Thanks,

Gordon Sim