You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by czy11421 <cz...@gmail.com> on 2009/09/18 21:11:18 UTC
what is best practices to implement Multithread + JMS ?
what is best practices to implement Multithread + JMS ?
Let us assume we are listening financial market data feed. I have this
coding like this:
class MyClass implements Listener{
private final ExecutorService pool = Executors.newFixedThreadPool(poolSize);
onMessage(...){
pool.execute(new Handler(Message));
}
class Handler implements Runnable {
public run(){
// get data, same it into db
}
}
}
Anybody can share your experiences in this topic ?
Thanks.
Re: what is best practices to implement Multithread + JMS ?
Posted by czy11421 <cz...@gmail.com>.
Bruce Snyder wrote:
> On Fri, Sep 18, 2009 at 8:42 PM, Robert Nicholson
> <ro...@gmail.com> wrote:
>
>> Also, Is there a simple example of a JMS consumer that's using DMLC with
>> ActiveMQ available?
>>
>
> I'm attaching a zip file that contains some very basic examples of
> using the DefaultMessageListenerContainer. These examples currently
> don't make use of task executors or transactions.
>
> I'm CC'ing you directly to make sure that they make it to you as I
> don't recall if the list allows attachments.
>
> Bruce
>
Appreciate your samples. These samples help me understand those
container. But the sample is using spring annotation, even I can't find
bean configuration file. So I try to stay traditional way, and it works.
If anybody requests it, I am glad to upload it.
Regards.
Re: what is best practices to implement Multithread + JMS ?
Posted by Bruce Snyder <br...@gmail.com>.
On Fri, Sep 18, 2009 at 8:42 PM, Robert Nicholson
<ro...@gmail.com> wrote:
> Also, Is there a simple example of a JMS consumer that's using DMLC with
> ActiveMQ available?
I'm attaching a zip file that contains some very basic examples of
using the DefaultMessageListenerContainer. These examples currently
don't make use of task executors or transactions.
I'm CC'ing you directly to make sure that they make it to you as I
don't recall if the list allows attachments.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: what is best practices to implement Multithread + JMS ?
Posted by Robert Nicholson <ro...@gmail.com>.
Also, Is there a simple example of a JMS consumer that's using DMLC
with ActiveMQ available?
Re: what is best practices to implement Multithread + JMS ?
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, Sep 22, 2009 at 7:42 AM, Robert Nicholson
<ro...@gmail.com> wrote:
> I know of no way to only acknowledge what you've processed without
> potentially acknowledging what you're might not have already processed.
>
> In order to ensure that all records I have received are processed I
> currently store the state in the database such that if my process needs to
> be
> restarted without having processed what's been removed from the queue it can
> consult the database to determine what's left to do.
>
> This obviously is a very expensive operation and I'm looking for ideas as to
> how to avoid this?
>
> One way I found was to not do the processing in a separate thread. IF I
> dispatch to "child queues" I can then have a transacted session and
> have consumers/receivers attached to those queues such that I know when I've
> committed the session I've processed what was received. I believe
> it's still possible to commit messages that weren't processed but this seems
> to have the semantics where if I rolled back the session the messages
> are still left on the queue.
>
> When you introduce a thread pool and do the work in worker threads it then
> becomes challenging to get the right semantics where you know
> that if something hasn't been processed but a delivery attempt was made that
> that will still be on the queue. JMS doesn't let me do anything
> with the Session in a worker thread except close it. I cannot commit or
> acknowledge in a worker thread. ie. the place where I know I've fully
> processed the message or not.
>
> Looking for ideas.
Oh, are you talking about how calling message.acknowledge() acks all
messages on the session rather than just a single message? Starting in
ActiveMQ 5.2.0, a new ack mode was added named INDIVIDUAL_ACKNOWLEDGE
that is used to ack messages individually:
http://activemq.apache.org/maven/activemq-core/apidocs/org/apache/activemq/ActiveMQSession.html#INDIVIDUAL_ACKNOWLEDGE
Does this help your scenario?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: what is best practices to implement Multithread + JMS ?
Posted by Robert Nicholson <ro...@gmail.com>.
I know of no way to only acknowledge what you've processed without
potentially acknowledging what you're might not have already processed.
In order to ensure that all records I have received are processed I
currently store the state in the database such that if my process
needs to be
restarted without having processed what's been removed from the queue
it can consult the database to determine what's left to do.
This obviously is a very expensive operation and I'm looking for ideas
as to how to avoid this?
One way I found was to not do the processing in a separate thread. IF
I dispatch to "child queues" I can then have a transacted session and
have consumers/receivers attached to those queues such that I know
when I've committed the session I've processed what was received. I
believe
it's still possible to commit messages that weren't processed but this
seems to have the semantics where if I rolled back the session the
messages
are still left on the queue.
When you introduce a thread pool and do the work in worker threads it
then becomes challenging to get the right semantics where you know
that if something hasn't been processed but a delivery attempt was
made that that will still be on the queue. JMS doesn't let me do
anything
with the Session in a worker thread except close it. I cannot commit
or acknowledge in a worker thread. ie. the place where I know I've fully
processed the message or not.
Looking for ideas.
On Sep 20, 2009, at 12:11 PM, Bruce Snyder wrote:
> On Fri, Sep 18, 2009 at 8:42 PM, Robert Nicholson
> <ro...@gmail.com> wrote:
>> I've been looking at DMLC lately.
>>
>> Acknowledge semantics are a little different when using it though?
>
> There are four types of acknowledgement in the message listener
> container as listed in the AbstractMessageListenerContainer. I'd
> copy/paste the acknowledgement modes here but the Javadoc is kinda big
> but it's comprehensive. Take a look here:
>
> http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/listener/AbstractMessageListenerContainer.html
>
>> Isn't there the notion that you acknowledge before dispatch and
>> therefore
>> there
>> lies an opportunity to acknowledge something from the queue and not
>> actually
>> process it
>> because of an outage.
>
> Acknowledgement semantics depend on the mode you choose. Here are some
> items that tend to confuse:
>
> 1) "sessionAcknowledgeMode" set to "AUTO_ACKNOWLEDGE" (default):
> Automatic message acknowledgment before listener execution; no
> redelivery in case of exception thrown.
>
> 2) "sessionAcknowledgeMode" set to "CLIENT_ACKNOWLEDGE": Automatic
> message acknowledgment after successful listener execution; no
> redelivery in case of exception thrown.
>
> 3) "sessionAcknowledgeMode" set to "DUPS_OK_ACKNOWLEDGE": Lazy message
> acknowledgment during or after listener execution; potential
> redelivery in case of exception thrown.
>
> 4) "sessionTransacted" set to "true": Transactional acknowledgment
> after successful listener execution; guaranteed redelivery in case of
> exception thrown.
>
>> JMS doesn't let you commit the session in any other thread than the
>> thread
>> of control
>> of the session so you aren't going to have a guarantee that you
>> actually
>> processed
>> what was acknowledged/committed in the Session.
>
> Again, that depends on the ack mode. See above.
>
>> Also, DMLC doesn't do anything for you regarding order of
>> processing so how
>> does
>> Spring address that problem? Am I expected to have N's DMLC's and
>> dispatch
>> based
>> on my own rules rather than something that can share from a common
>> thread
>> pool but
>> still guarantee some order of processing?
>
> Controlling the order in which messages are processed on the
> client-side is not a goal of the Spring message listener container
> architecture. In a scenario where the order of processing is
> important, you would only want to have a single consumer. Either that
> or you'd need to wholly separate the consumption/ack stage from the
> processing stage. Still, you'd need some additional strategy for such
> a restriction. Please explain the use case you have in mind for
> controlling the order in which messages are processed.
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!
> G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
Re: what is best practices to implement Multithread + JMS ?
Posted by Bruce Snyder <br...@gmail.com>.
On Fri, Sep 18, 2009 at 8:42 PM, Robert Nicholson
<ro...@gmail.com> wrote:
> I've been looking at DMLC lately.
>
> Acknowledge semantics are a little different when using it though?
There are four types of acknowledgement in the message listener
container as listed in the AbstractMessageListenerContainer. I'd
copy/paste the acknowledgement modes here but the Javadoc is kinda big
but it's comprehensive. Take a look here:
http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/listener/AbstractMessageListenerContainer.html
> Isn't there the notion that you acknowledge before dispatch and therefore
> there
> lies an opportunity to acknowledge something from the queue and not actually
> process it
> because of an outage.
Acknowledgement semantics depend on the mode you choose. Here are some
items that tend to confuse:
1) "sessionAcknowledgeMode" set to "AUTO_ACKNOWLEDGE" (default):
Automatic message acknowledgment before listener execution; no
redelivery in case of exception thrown.
2) "sessionAcknowledgeMode" set to "CLIENT_ACKNOWLEDGE": Automatic
message acknowledgment after successful listener execution; no
redelivery in case of exception thrown.
3) "sessionAcknowledgeMode" set to "DUPS_OK_ACKNOWLEDGE": Lazy message
acknowledgment during or after listener execution; potential
redelivery in case of exception thrown.
4) "sessionTransacted" set to "true": Transactional acknowledgment
after successful listener execution; guaranteed redelivery in case of
exception thrown.
> JMS doesn't let you commit the session in any other thread than the thread
> of control
> of the session so you aren't going to have a guarantee that you actually
> processed
> what was acknowledged/committed in the Session.
Again, that depends on the ack mode. See above.
> Also, DMLC doesn't do anything for you regarding order of processing so how
> does
> Spring address that problem? Am I expected to have N's DMLC's and dispatch
> based
> on my own rules rather than something that can share from a common thread
> pool but
> still guarantee some order of processing?
Controlling the order in which messages are processed on the
client-side is not a goal of the Spring message listener container
architecture. In a scenario where the order of processing is
important, you would only want to have a single consumer. Either that
or you'd need to wholly separate the consumption/ack stage from the
processing stage. Still, you'd need some additional strategy for such
a restriction. Please explain the use case you have in mind for
controlling the order in which messages are processed.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: what is best practices to implement Multithread + JMS ?
Posted by Robert Nicholson <ro...@gmail.com>.
I've been looking at DMLC lately.
Acknowledge semantics are a little different when using it though?
Isn't there the notion that you acknowledge before dispatch and
therefore there
lies an opportunity to acknowledge something from the queue and not
actually process it
because of an outage.
JMS doesn't let you commit the session in any other thread than the
thread of control
of the session so you aren't going to have a guarantee that you
actually processed
what was acknowledged/committed in the Session.
Also, DMLC doesn't do anything for you regarding order of processing
so how does
Spring address that problem? Am I expected to have N's DMLC's and
dispatch based
on my own rules rather than something that can share from a common
thread pool but
still guarantee some order of processing?
On Sep 18, 2009, at 4:55 PM, Bruce Snyder wrote:
> On Fri, Sep 18, 2009 at 1:11 PM, czy11421 <cz...@gmail.com> wrote:
>> what is best practices to implement Multithread + JMS ?
>>
>> Let us assume we are listening financial market data feed. I have
>> this
>> coding like this:
>>
>> class MyClass implements Listener{
>> private final ExecutorService pool =
>> Executors.newFixedThreadPool(poolSize);
>> onMessage(...){
>> pool.execute(new Handler(Message)); }
>> class Handler implements Runnable {
>> public run(){
>> // get data, same it into db
>> }
>> }
>> }
>>
>> Anybody can share your experiences in this topic ?
>
> When building JMS consumers, I recommend the use of the Spring
> DefaultMessageListenerContainer:
>
> http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/listener/DefaultMessageListenerContainer.html
>
> The DMLC provides the ability to set a TaskExecutor, a range of
> concurrent consumers, various tiered caching levels (connection,
> session, consumer), it works with transactions and much more. The
> reason I recommend the DMLC (or one of the other message listener
> containers) is because writing JMS clients is a lot of work and the
> Spring message listener containers dramatically reduce the complexity
> thereby saving you quite a lot of time. Given the tremendous
> flexibility, the robustness, the high amount of configurability and
> the widespread deployment in businesses all over the world (including
> in the financial markets), there really no reason not to use it.
>
> Together with the Spring CachingConnectionFactory
> (http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/connection/CachingConnectionFactory.html
> ),
> these tools will speed your JMS development client developer markedly.
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!
> G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
Re: what is best practices to implement Multithread + JMS ?
Posted by Bruce Snyder <br...@gmail.com>.
On Fri, Sep 18, 2009 at 1:11 PM, czy11421 <cz...@gmail.com> wrote:
> what is best practices to implement Multithread + JMS ?
>
> Let us assume we are listening financial market data feed. I have this
> coding like this:
>
> class MyClass implements Listener{
> private final ExecutorService pool =
> Executors.newFixedThreadPool(poolSize);
> onMessage(...){
> pool.execute(new Handler(Message)); }
> class Handler implements Runnable {
> public run(){
> // get data, same it into db
> }
> }
> }
>
> Anybody can share your experiences in this topic ?
When building JMS consumers, I recommend the use of the Spring
DefaultMessageListenerContainer:
http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/listener/DefaultMessageListenerContainer.html
The DMLC provides the ability to set a TaskExecutor, a range of
concurrent consumers, various tiered caching levels (connection,
session, consumer), it works with transactions and much more. The
reason I recommend the DMLC (or one of the other message listener
containers) is because writing JMS clients is a lot of work and the
Spring message listener containers dramatically reduce the complexity
thereby saving you quite a lot of time. Given the tremendous
flexibility, the robustness, the high amount of configurability and
the widespread deployment in businesses all over the world (including
in the financial markets), there really no reason not to use it.
Together with the Spring CachingConnectionFactory
(http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/jms/connection/CachingConnectionFactory.html),
these tools will speed your JMS development client developer markedly.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder