Adding more controls to the current WS-RM component

[Aki Yoshida <el...@googlemail.com>]

Hi,
I have been using the CXF WS-RM component for a while. And I can say,
I am enjoying it. Right now, among others, I am experimenting with
some ideas to get more control over its behavior and more robustness
in certain situation. I would like to share these ideas. In
particular, I am describing the three related issues whose possible
solution may affect the current RMStore API and wanted to ask your
opinion before CXF 2.4.0 comes out.

1. Retry Behavior at Source Sequence
Currently, there is no maximum retry count and a retry can run forever
if the message is not acknowledged. I would like to introduce an
optional parameter to configure the maximum retry count. If the retry
count reaches this limit, the status of the message can be set to
undelivered or failed. The failed message can be removed from the
retransmission queue and may only be retried after a manual
intervention.

For this change, it is necessary to introduce this property itself and
this is simple. In addition, it is to decide whether to store the
status information of the message in the storage. In this case, the
retry count information becomes persistent. I think we can omit
storing this count information per message as it incurs increased
access to the storage. However, if all the unacknowledged messages in
a sequence are set failed, the sequence should be marked as failed in
the storage so that they can be excluded from being restarted at the
next start up.

2. SourceSequence's Expire handling
This field is stored in the source sequence but it does not seem to be
used during the restart of the endpoints to decide which messages must
be placed in the retransmission queue. This value along with the
sequence status value described in 1 should be used to control the
retry behavior of the old unacknowledged messages.

3. Sequence caching in memory
Currently, the source and destination sequences are loaded into memory
from the storage at the startup of the endpoints. This may consume an
increased amount of unnecessary memory if many of the sequences are
not active while not yet expired. This situation may happen when
several conditions are satisfied, for example, when the sequence
termination policy configured to limit the number of unacknowledged
messages per sequence, occurring frequent transmission errors, and
messages generated at a high rate.

For the source sequence, using the change suggested for point 1 and 2,
we can decrease the number of active source sequences. In this case,
only those active source sequences can be read from the storage and
their associated unacknowledged messages can be placed in the
retransmission queue.

For the destination sequence, the situation is different (as there is
no retry) and some no-expiration-set sequences may remain alive
although they may be used only infrequently or perhaps not at all. The
good news, however, is that we need the sequence only when a message
with that particular sequence ID arrives. In this case, some fixed
number of destination sequences can be held in memory and a
destination sequence can be looked up in the storage upon this cache
miss.

To achieve this on-demand loading of the destination sequences, I
added in RMStore a new method to retrieve a single destination
sequence instance given its sequence ID. In addition, considering
points 1 and 2, I would like to have sequence retrieval methods to
retrieve the expired or failed sequences. Here, I am not sure how I
should name them (or rather use an additional filter parameter to the
getSourceSequences method). In either case, this affects the RMStore
interface unless we opt for introducing another API, I wanted to get
your suggestion before 2.4 comes out.

I would greatly appreciate your suggestion.

Best regards, Aki

Re: Adding more controls to the current WS-RM component

[Aki Yoshida <el...@googlemail.com>]

2011/3/17 Dennis Sosnoski <dm...@sosnoski.com>:
> On 03/17/2011 12:40 PM, Aki Yoshida wrote:
>>
>> ...
>>
>> 1. Retry Behavior at Source Sequence
>> Currently, there is no maximum retry count and a retry can run forever
>> if the message is not acknowledged. I would like to introduce an
>> optional parameter to configure the maximum retry count. If the retry
>> count reaches this limit, the status of the message can be set to
>> undelivered or failed. The failed message can be removed from the
>> retransmission queue and may only be retried after a manual
>> intervention.
>
> I don't see any way of making this work as stated, since it would violate
> the WS-RM specification (which doesn't allow the sender to arbitrarily
> decide to drop individual messages). What *could* be done is to add maximum
> retry count and maximum retry times, and abort the whole sequence if these
> are exceeded. But you'd need to somehow notify the client when this occurs.
> Any thoughts on that?

I didn't mean to violate the WS-RM spec in the protocol level. I meant
exactly the option that you say it could. These non-deliverable
messages need to be marked as failed and we need an option to manually
intervene with the next step, either restart them again or terminate
them permanently. The client can find out using the current callback
API to determine whether the message has been delivered or not. We can
provide some JMS beans to check the statuses and to provide this
manual intervension option.

>
>> ...
>> 3. Sequence caching in memory
>> Currently, the source and destination sequences are loaded into memory
>> from the storage at the startup of the endpoints. This may consume an
>> increased amount of unnecessary memory if many of the sequences are
>> not active while not yet expired. This situation may happen when
>> several conditions are satisfied, for example, when the sequence
>> termination policy configured to limit the number of unacknowledged
>> messages per sequence, occurring frequent transmission errors, and
>> messages generated at a high rate.
>
> WS-RM implementations *should* terminate the sequence when they're done
> using it, so a big build up of unexpired sequences should not occur. But I'd
> think the memory usage by these sequences is relatively insignificant
> anyway. Have you seen cases where there are really large numbers of
> sequences open and this becomes a problem?

The current policy configuraiton allows restricting the number of
unacknowledged messages per sequence. So if there are massive
transmission errors ocurring, while the client continues to send
messages, sequences may build up. I haven't observed this problem but
this could become a possible risk in a real productive environment. We
may also need an option to block the client call or throw an exception
if the sequence or message backlog is building up rapidly and
consuming too much resources.

Regards, Aki

Re: Adding more controls to the current WS-RM component

[Dennis Sosnoski <dm...@sosnoski.com>]

On 03/17/2011 12:40 PM, Aki Yoshida wrote:
> ...
>
> 1. Retry Behavior at Source Sequence
> Currently, there is no maximum retry count and a retry can run forever
> if the message is not acknowledged. I would like to introduce an
> optional parameter to configure the maximum retry count. If the retry
> count reaches this limit, the status of the message can be set to
> undelivered or failed. The failed message can be removed from the
> retransmission queue and may only be retried after a manual
> intervention.

I don't see any way of making this work as stated, since it would 
violate the WS-RM specification (which doesn't allow the sender to 
arbitrarily decide to drop individual messages). What *could* be done is 
to add maximum retry count and maximum retry times, and abort the whole 
sequence if these are exceeded. But you'd need to somehow notify the 
client when this occurs. Any thoughts on that?

> ...
> 3. Sequence caching in memory
> Currently, the source and destination sequences are loaded into memory
> from the storage at the startup of the endpoints. This may consume an
> increased amount of unnecessary memory if many of the sequences are
> not active while not yet expired. This situation may happen when
> several conditions are satisfied, for example, when the sequence
> termination policy configured to limit the number of unacknowledged
> messages per sequence, occurring frequent transmission errors, and
> messages generated at a high rate.

WS-RM implementations *should* terminate the sequence when they're done 
using it, so a big build up of unexpired sequences should not occur. But 
I'd think the memory usage by these sequences is relatively 
insignificant anyway. Have you seen cases where there are really large 
numbers of sequences open and this becomes a problem?

   - Dennis