Runtime isolation of Message Processors

[shafreen anfar <an...@gmail.com>]

Hi All,

At the moment, even though it looks like we are using different quartz
schedulers for each Message Processor, we are not. In other words, all the
Message Processors re-use the same quartz scheduler. Following is how it
looks in the runtime.

"*DefaultQuartzScheduler_QuartzSchedulerThread*" #45 prio=5 os_prio=0
tid=0x00007f8464af1000 nid=0x7e3d in Object.wait() [0x00007f83d36f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:301)
- locked <0x00000000f73612e0> (a java.lang.Object)

"*DefaultQuartzScheduler_Worker-10*" #44 prio=5 os_prio=0
tid=0x00007f8464aed800 nid=0x7e3c in Object.wait() [0x00007f83d37f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
- locked <0x00000000f7279078> (a java.lang.Object)

"*DefaultQuartzScheduler_Worker-9*" #43 prio=5 os_prio=0
tid=0x00007f8464aec000 nid=0x7e3b in Object.wait() [0x00007f83d38f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
- locked <0x00000000f7278b78> (a java.lang.Object)
*...*

What really happens is, quartz creates one scheduler with ten worker
threads as per the default behaviour. Therefore if something goes wrong in
one Message Processor it could result in affecting all the Message
Processors. Of course, quartz has a way to make sure one message processor
always gets only one thread. But in case, say, a developer writing a new
Message Processor forgot to configure this, that particular Message
Processor could affect all the other Message Processors. Also, in runtime,
when there is a issue, it will be very hard isolate which Message Processor
actually causing the issue.

Therefore, I am proposing to isolate Message Processors completely during
the runtime as well. Which makes Message Processors more resilient as this
is sort of similar to bulkheading and less vulnerable to developer
mistakes. With the suggested change runtime would look like below for a
given Message Processor. Please note that in this case, my Massage
Processor name is *Processor3.*

"Processor3_QuartzSchedulerThread" #41 prio=5 os_prio=0
tid=0x00007ff95cbbd800 nid=0xf94 in Object.wait() [0x00007ff8cbdfc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f7db09e0> (a java.lang.Object)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:301)
- locked <0x00000000f7db09e0> (a java.lang.Object)

"Processor3_Worker-1" #40 prio=5 os_prio=0 tid=0x00007ff95cbbc800 nid=0xf93
in Object.wait() [0x00007ff8cbefd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f7db0230> (a java.lang.Object)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555)
- locked <0x00000000f7db0230> (a java.lang.Object)

WDYT ?