Recent work on clustering

[Afkham Azeez <af...@gmail.com>]

Hi Folks,
You may have noticed that there has been some new work going on in the Axis2
clustering implementation. Here is a summary of what has been taking place
in the recent past:

1. Introduced RpcChannel
-------------------------------------
This is a blocking request-response communication channel. When an Axis2
node starts up, it needs to get the latest state and configuration from a
neighbor. Until this information is received, the node cannot start service,
hence should block until these messages are received. A Tribes RpcChannel is
ideal for this scenario.


2. Introduced SYNC_ACK & ASYNC mode transmissions at the Tribes layer
--------------------------------------------------------------------------------------------------------------
Earlier, we were handling ACKs at the Axis2 layer, not the Tribes layer.
This approach had a significant overhead since the ACK messages have to be
serialized/deserialized. Now we do this at the Tribes layer using SYNC_ACK
transmission. In this scenario, the channel.send blocks until all members
have successfully processed a message and sent an ACK. If a member  does not
receive an ACK within a specified period, Tribes will do a restransmission.
The maximum number of retries can be configured in the axis2.xml cluster
initialization section (<parameter name="maxRetries">5</parameter>).  As you
may be already thinking, this mode of communication will have a hit on the
response times since the channel has to block until all members ACK. This
may be undesirable and unnecessary in some circumstances. We can use ASYNC
mode of transmission to speed up things. In this case, as soon as the sender
sends the message down to its OS's TCP buffer, it returns.  Switching
between SYNC_ACK & ASYNC modes can be done by changing the "synchronizeAll"
clustering parameter in the axis2.xml (<parameter
name="synchronizeAll">true</parameter>). After this change, I was able to
observe a significant performance improvement.


3. Introduced AtMostOneInterceptor
----------------------------------------------------
As mentioned in #2 above, Tribes takes care of retransmissions, which can
lead to the same message being received more than once by a receiver. To
handle this, we have introduced an AtMostOnceInterceptor, which intercepts
messages on the channel on the receiver, and drops duplicate messages.

4. Introduced OrderInterceptor
--------------------------------------------
This interceptor guarantees that sender order is preserved. i.e. each
correct receiver in the cluster will receive the messages in the same order
that they were sent by the sender. When sending a message, we set a special
bit in the Channel Options, and all messages that have this bit set will be
ordered.

5. Introduced Membership awareness
------------------------------------------------------
We do not do channel.getMembers anymore. A Tribes membership listener
notifies the node about view changes (new members joining and members
leaving the group), and the member list is maintained on each node. This
way, all knowledge about the members is in one place. Also, when we try to
get the state and configuration information, we first try to get it from the
longest living member first (assuming that this member will have all the
relevant information) failing which, we try a random member.

6. Using ByteMessages
-----------------------------------
Now we send/receive Tribes ByteMessages on the channel, instead of
java.ioserialized messages. When this message is used, Tribes does not
try to
serialize/deserialize the messages, and sends raw bytes, which has a
performance advantage. On the receiving side, we handle the deserialization
at the Axis2 layer, if necessary. Also this way, redundant messages &
messages not intended for this group do not get deserialized.

7. Ability to provide custom classloaders
-----------------------------------------------------------
Earlier, we were able to send across only objects that can be created by
classes loaded from the root classloader. Hence custom POJOs that are
contained within AARs & MARs cannot be sent across. We've changed the code
so that now we can get hold of the relevant classloaders and deserialize
these POJOs.

8. Improved test suite
--------------------------------
These are integration tests. The tests fire-up two tribes nodes. Then
context replication tests are carried out.

Please send you feedback & valuable suggestion on these changes.

-- 
Thanks
Afkham Azeez

http://www.wso2.org
http://azeez78.blogspot.com
GPG Fingerprint: 643F C2AF EB78 F886 40C9  B2A2 4AE2 C887 665E 0760