SOAPMonitor single point of failure

[Alex Burmester <ad...@beast.com>]

Hi I'm wondering if any of the developers or someone who has
seen this happen might want to comment.

Yesterday I saw SOAPMonitor bring a pretty high volumne site to a halt.
Here's what I think happened.

SOAPMonitor was used to check something on a number of the production 
machines.  I believe it was left running. (not a good idea, I know)  
About 4 hours later our application servers stopped responding.  A thread 
dump showed that all the threads were waiting on the SOAPMonitor.
I'm guessing that the SOAPMonitor applet had been left running and at some 
point had some type of a problem like running out of memory or the vm just 
got hosed or something.  From looking at the source it seems that the
SOAPMonitorHandler will keep pumping messages via the SOAPMonitorService
to the ConnectionThreads and try to send them out the wire.  It looks like
if something gets hung up with the socket from the applet that it might 
cause what we saw which is that threads get hung waiting for 
publishMessage to return.

Anyone seen this or have any ideas how SOAPMonitor could be improved to 
cope with this situation? (perhaps a tuneable socket timeout? or some 
async decoupling of the Handler from the sending of the messages to the 
applet?)

Thanks,

Alex.