svn commit: r781882 - /activemq/sandbox/activemq-flow/webgen/src/architecture.page

[cm...@apache.org]

Author: cmacnaug
Date: Fri Jun  5 01:18:41 2009
New Revision: 781882

URL: http://svn.apache.org/viewvc?rev=781882&view=rev
Log:
Updated architecture notes.

Modified:
    activemq/sandbox/activemq-flow/webgen/src/architecture.page

Modified: activemq/sandbox/activemq-flow/webgen/src/architecture.page
URL: http://svn.apache.org/viewvc/activemq/sandbox/activemq-flow/webgen/src/architecture.page?rev=781882&r1=781881&r2=781882&view=diff
==============================================================================
--- activemq/sandbox/activemq-flow/webgen/src/architecture.page (original)
+++ activemq/sandbox/activemq-flow/webgen/src/architecture.page Fri Jun  5 01:18:41 2009
@@ -1,11 +1,136 @@
-
 ---
 title: Architectural Overview
 --- pipeline:textile
 
-h2. Overview
+h1. Overview
+
 This page explores some of the details around core components. 
 
-h2. Flow Control
 
-TODO: I've got some more background notes that I need to 
+h1. Getting Started:
+
+Some pointers to get you going with the prototype:
+
+It is located in the activemq sandbox at: "https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow":https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow}
+
+Running some testcases:
+
+When running these testcases you should use the -server jvm argument:
+
+# *activemq-queue module org.apache.activemq.flow.MockBrokerTest*. This is a good starting point for looking at some of the flow control, dispatcher and queue modules. It is a very basic MockBroker which uses a lightweight proto-buf based protocol. You should feel free to play around with some of the hardcoded options in the test (e.g. ptp=true, tcp=true, threadsPerDispatcher etc).
+# *activemq-openwire module org.apache.activemq.broker.SharedQueuePerfTest.* This test does performance testing against the SharedQueue implementation and is a good place to start looking at queue persistence and the CursoredQueue. It employs openwire messages with using network connecitons. Check out *org.apache.activemq.broker.BrokerQueueStore* which currently hardcodes the queue sizes; you can change DEFAULT_SHARED_QUEUE_PAGING_THRESHOLD and the associated persistence policy to play around with paging.
+# *activemq-all module org.apache.activemq.broker.openwire.OpenWireBrokerTest*. This test uses the full blown broker, adds in a tcp transport layert and provides options for persistence and durable subscriptions (albeit asynchronous at the moment). It extends *org.apache.activemq.broker.BrokerTest* in the activemq-broker module which has some additional hard coded options to play around with. Again refer to *org.apache.activemq.broker.BrokerQueueStore* to mess around with queue sizes.
+
+
+h1. Flow Control
+
+The activemq-flow package is meant to be a standalone module that deals generically with Resources and Flow's of elements that flow through and between them. Each resource creates a FlowController for each of it's Flows which is assigned a corresponding FlowLimiter. As elements (e.g. messages) pass from one resource to another they are passed through the downstream resource's FlowController which updates its Limiter. If propagation of an element from one resource to another causes the downstream limiter to become throttled the associated FlowController will block the source of the element. The flow module is used heavily by the rest of the core for memory and disk management.
+
+* _Memory Mangement:_ Memory is manged based on the resources in play \-\- the usage is computed by summing of the space allocated to each of the resources' limiters. This strategy intentionally avoids a centralized memory limit which leads to complicated logic to track when a centralized limiter needs to be decremented and avoids contention between multiple resources/threads accessing the limiter and also reduces the potential for memory limiter related deadlocks. However, it should be noted that this approach doesn't preclude implementing centralized limiters in the future.
+* _Flow Control:_ As messages propagate from one resource A to another B, then if A overflows B's limit, B will block A and A can't release it's limiter space until B unblocks it. This allowance for overflow into downstream resources is a key concept in flow control performance and ease of use. Provided that the upstream resource has already accounted for the message's memory it can freely overflow any downstream limiter providing it reserves space from elements that caused overflow.
+* _Threading Model:_ Note that as a message propagates from A to B, that the general contract is that A won't release it's memory if B blocks it during the course of dispatch. This means that it is not safe to perform a thread handoff during dispatch between two resources since the thread dispatching A relies on the message making it to B (so that B can block it) prior to A completing dispatch.
+* _Management/Visibility:_ Another intended use of the activemq-flow module is to assist in visibility e.g. provide an underlying map of resources that can be exposed via tooling to see the relationships between sources and sinks of messages and to find bottlenecks ... this aspect has been downplayed for now as we have been focusing more on the queueing/memory management model in the prototype, but eventually the flow package itself will provide a handy way of providing visibility in the system particularly in terms of finding performance bottlenecks.
+
+*FlowResource (FlowSink and FlowSource):* A container for FlowControllers providing some lifecycle related logic. The base resource class handles interaction/registration with the FlowManager (below).
+
+*FlowManager:* Registry for Flow's and FlowResources. The manager will provide some hooks into system visibility. As mentioned above this aspect has been downplayed somewhat for the present time.
+
+*FlowController*: Wraps a FlowLimiter and actually implements block/resume logic
+
+*FlowLimiter*: Defines the limits enforced by a FlowController. Currently the package has size based limiter implementations, but eventually should also support other common limiter types such as rate based limiters. The limiter's are also extended at other points in the broker (for example implementing a protocol based WindowLimiter). It is also likely that we would want to introduce CompositeLimiters to combine various limiter types.
+
+*Flow*: The concept of a flow is not used very heavily right now. But a Flow defines the stream of elements that can be blocked. In general the prototype creates a single flow per resource, but in the future a source may break it's elements down into more granular flows on which downstream sinks may block it. One case where this is anticipated as being useful is in networks of brokers where-in it may be desirable to partition messages into more granular flows (e.g based on producer or destination) to avoid blocking the broker-broker connection uncessarily).
+
+h1. Dispatcher / Threading Model
+
+The dispatcher package is basically a smart thread pool \-\- at the new broker core almost everything should be driven by a dispatcher to keep thread count down and scalability up.
+
+    * Components in the core create DispatchContexts which they use to request dispatch via a Dispatcher. The dispatch attempts to be smart about grouping together dispatch contexts that interact with one another to reduce contended synchronization. For example if we have a producer->queue->consumer scenario each with their own dispatch context, the dispatcher attempts to group them all on the same thread to avoid contention.
+    * The Dispatcher is priority based which in conjunction with partitioned queues assists in dispatching higher priority messages
+    * The Dispatcher also support creation of simple Executors that operate on the dispatch threads.
+    * Supports timer callbacks.
+
+h1. Persistence Store
+
+KahaDB based. As it stands now there is a single worker thread / queue that drives the store impl (BrokerDatabase).
+
+    * Like other resources, the BrokerDatabase employs a flow controller so as elements are added to the queue they may block the source (e.g. a producer) if the configured memory limit is exceeded.
+    * The BrokerDatabase allows events to be delayed. This is at a key optimization for persistent delivery (in conjunction with the CursoredQueue implementation). When a persistent message is placed on a queue with consumers that are keeping up, it is placed on the DB queue with a flush delay. If the consumer, acknowledged the message prior to the delay expiring it is removed from the queue and doesn't have to hit the disk.
+
+The approach of a single worker thread driving the store is still under debate. It's pros are that it reduces contention at the store layer, allows batching of several operations into a single disk sync, and simplifies ordering of interrelated db operations. The major downside is that it doesn't allow the store to leverage all cpus for encoding decoding work. The store implementation itself is not yet complete and needs more work.
+
+h1. Queues
+
+The activemq-queue module contains basic queue implementations and a Subscription interfaces. It relies heavily on activemq-flow and activemq-dispatcher modules. The intent is to allow the queue module to be used by both the broker and the client. The queue module has 3 basic queue types:
+* Exclusive Queue: Only one subscription is attached to the queue (e.g. topic subscriber or durable)
+* Share Queue: A load balanced queue where messages are loadbalanced across multiple consumers.
+* Relay: Like a queue but simply a pass through (useful in cases where it doesn't make sense to buffer up messages but where you still want to a FlowController to throttle sources.
+
+In addition the queue module defines partitioned queues to allow segmenting messages based on arbitrary criteria (more below). It also has priority based implementations built in.
+
+
+h2. CursoredQueue
+
+The hope is to use the CursoredQueue as the single persistence mechanism for messages, and serves as the basis for other queue types that require persistence. It supports browsing and expiration.
+It the case of SharedQueues, Durables, Transactions and even non durables that wish to offload messages to the store when the consumer is too slow.
+
+* Transactions: usecase is to hold messages in memory until commit time, spooling to disk if transaction memory limit is reached.
+* SharedQueue store queue messages.
+* DurableQueue store durable messsages
+* TopicSubscription when configured to
+
+It can operate in a couple of different modes:
+
+* Keep all elements in memory and throttle sources to the memory limit \-\- persisting only elements whose QoS level dictates it.
+* Two paging modes otherwise. Offload the message itself but keep a placeholder in memory (increases performance), or page out place holders to allow for very large queue backlogs (where the placeholders themselves become a memory burden).
+
+There is still a fair amount of policy work /customization to be done here with respect to paging queues, particular with respect to slow or disconnected subscribers:
+
+* Expunge Policy: When to discard messages (if ever)
+* Handling of temporary bursts: In general if subscribers can keep up with the publishers it is best to keep messages in memory instead of paging them out, however, for the case of a temporary burst paging to disk can be appropriate
+* When paging out page out a reduced rate or as fast as possible?
+* Disconnected Subscriber (no subs on queue or disconnected durable): throttle the rate at which producers send messages to the store to improve recovery time?
+* Other usecases/scenarios?
+
+
+h2. Partitioning of queues
+
+We've also introduced the concept of partitioned queue in which a single queue can be partitioned to increase parallelization of queue processing. Additionally:
+
+* Priority based partitions allow dispatch of higher priority messages first.
+* Can be used for JMSXGroupID (not implemented yet)
+* For clustering connections partitioning can be used to create parallel independently limited message flows between brokers (so if there is a slow consumer at the other side only only the streams of messages being sent to it will be blocked).
+
+h1. Work In Progress
+
+
+h2. Transactions:
+
+* Should be backed by cursored queue
+
+h2. Large Message Support:
+
+* There are some stubs in the Store interface for this, but basically for large messages we should be able to stage them to disk, and when given to a consumer chunk the payload out to keep memory low. The cursored queue will need to be updated to handle this.
+
+h2. Protocol Support
+
+Lots of work still to be done here\!
+
+* Protocol Handlers are there for OpenWire and STOMP, and we'll want to add proto buf as well.
+* Do we convert things to a common message format on arrival or keep them in their native format doing transforms on the fly?
+* Common interfaces for Connection/Session/*Consumer for the broker to interact with?
+* Threading model ... We should probably come up with a good abstract base class for the protocol handlers that helps to provide a threading model that will make synchronization between I/O events (exceptions etc), protocol events and message dispatch). 
+
+h2. Replication:
+
+* This should be handled by KahaDB replication
+* Still need to think about replication of non persistent messages, though ...e.g. messages that are never saved to disk but only replicated to peers.
+* Recovery time. Keep state fairly hot to achieve fast failover time, we'll want to keep optimizations in this area in mind (basically messages that are in memory on the active broker are kept in memory on the standby as well).
+
+h2. General Interface cleanliness
+
+A fair amount of clean up work needs to be done to clean up and harden interfaces in the prototype. 
+
+h2. Testing
+
+As much of the work to date has been largely experimental in nature, testing up to now has focussed mainly on end to end type performance tests, but more unit testing and functional testing of edge cases such as slow subscriber test etc are needed. 
\ No newline at end of file