You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jason Dillon <ja...@planet57.com> on 2011/11/18 01:30:07 UTC
Overhead of component creation to send message
I'm wondering what sort of overhead there is to create and then close) the components needed to send a message, specifically after you have a started connection and using a vm:// transport.
I'm working on implementing distributed eventing for a server which already has its own eventing built-in (so adapting its events to JMS messages published to topics). The events can come from any thread and be sent to different topics based source event details. That seems to mean that for each local event I have to:
1) reference destination
2) create session
3) create producer
4) build message for event and send
5 ) close producer and session (discard destination)
#1 looks like its just object creation, but has some parsing of physical name (quite a few ops as it looks like)... so could potentially cache these (trade a bit of memory for a string lookup over always creating new instance)?
Not sure what overhead there is for #2, #3 or #5. Is there any documentation on roughly what these operations cost?
The destination + session could change so #3 would have to be done anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its better to have the local event handler queue up the publish in a BlockingQueue (or similar) so that a single thread + session (or potentially small pool of thread+session) could be used to a actually perform the publish?
Does anyone have any insight on to what would be best option for least overhead for this use-case?
--jason
Re: Overhead of component creation to send message
Posted by Jason Dillon <ja...@planet57.com>.
I did read that javadoc before, just wanted some additional assurance from the community that this was indeed accurate. Sounds like it is. Thanks :-)
--jason
On Dec 8, 2011, at 1:45 AM, Torsten Mielke wrote:
> From the Javadoc of PooledConnectionFactory:
>
> http://activemq.apache.org/maven/5.5.0/activemq-pool/apidocs/org/apache/activemq/pool/PooledConnectionFactory.html
>
>
>
> "A JMS provider which pools Connection, Session and MessageProducer instances so it can be used with tools like Camel and Spring's JmsTemplate and MessagListenerContainer. Connections, sessions and producers are returned to a pool after use so that they can be reused later without having to undergo the cost of creating them again. NOTE: while this implementation does allow the creation of a collection of active consumers, it does not 'pool' consumers. Pooling makes sense for connections, sessions and producers, which are expensive to create and can remain idle a minimal cost. Consumers, on the other hand, are usually just created at startup and left active, handling incoming messages as they come. When a consumer is complete, it is best to close it rather than return it to a pool for later reuse: this is because, even if a consumer is idle, ActiveMQ will keep delivering messages to the consumer's prefetch buffer, where they'll get held until the consumer is active again. If you are creating a collection of consumers (for example, for multi-threaded message consumption), you might want to consider using a lower prefetch value for each consumer (e.g. 10 or 20), to ensure that all messages don't end up going to just one of the consumers. See this FAQ entry for more detail: http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html "
>
> Hope this clarifies things.
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>
>
> On Dec 8, 2011, at 6:10 AM, Jason Dillon wrote:
>
>> Does the activemq-pool stuff cope with pooling connections, sessions and producers? Such that a component could access create and use these normally + close them, and under the covers activemq-pool will do the smart thing and reuse/avoid-close? Consumers are not pooled in similar fashion?
>>
>> I believe ^^^ is the case, but I just wanted to confirm incase I'm misunderstanding something.
>>
>> --jason
>>
>>
>> On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
>>
>>> Hi Jason,
>>>
>>> those operations are costly and if your component must open/close it for
>>> every message it will affect performances. In those cases it is recommended
>>> to use pool connection factory which caches those object and improve
>>> performances.
>>>
>>> See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
>>> on this topic (in case of Spring)
>>>
>>> Regards
>>> --
>>> Dejan Bosanac - http://twitter.com/dejanb
>>> -----------------
>>> The experts in open source integration and messaging - http://fusesource.com
>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>> Blog - http://www.nighttale.net
>>>
>>>
>>> On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>>>
>>>> I'm wondering what sort of overhead there is to create and then close) the
>>>> components needed to send a message, specifically after you have a started
>>>> connection and using a vm:// transport.
>>>>
>>>> I'm working on implementing distributed eventing for a server which
>>>> already has its own eventing built-in (so adapting its events to JMS
>>>> messages published to topics). The events can come from any thread and be
>>>> sent to different topics based source event details. That seems to mean
>>>> that for each local event I have to:
>>>>
>>>> 1) reference destination
>>>> 2) create session
>>>> 3) create producer
>>>> 4) build message for event and send
>>>> 5 ) close producer and session (discard destination)
>>>>
>>>> #1 looks like its just object creation, but has some parsing of physical
>>>> name (quite a few ops as it looks like)... so could potentially cache these
>>>> (trade a bit of memory for a string lookup over always creating new
>>>> instance)?
>>>>
>>>> Not sure what overhead there is for #2, #3 or #5. Is there any
>>>> documentation on roughly what these operations cost?
>>>>
>>>> The destination + session could change so #3 would have to be done
>>>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>>>> better to have the local event handler queue up the publish in a
>>>> BlockingQueue (or similar) so that a single thread + session (or
>>>> potentially small pool of thread+session) could be used to a actually
>>>> perform the publish?
>>>>
>>>> Does anyone have any insight on to what would be best option for least
>>>> overhead for this use-case?
>>>>
>>>> --jason
>>
>
>
>
>
>
Re: Overhead of component creation to send message
Posted by Torsten Mielke <to...@fusesource.com>.
From the Javadoc of PooledConnectionFactory:
http://activemq.apache.org/maven/5.5.0/activemq-pool/apidocs/org/apache/activemq/pool/PooledConnectionFactory.html
"A JMS provider which pools Connection, Session and MessageProducer instances so it can be used with tools like Camel and Spring's JmsTemplate and MessagListenerContainer. Connections, sessions and producers are returned to a pool after use so that they can be reused later without having to undergo the cost of creating them again. NOTE: while this implementation does allow the creation of a collection of active consumers, it does not 'pool' consumers. Pooling makes sense for connections, sessions and producers, which are expensive to create and can remain idle a minimal cost. Consumers, on the other hand, are usually just created at startup and left active, handling incoming messages as they come. When a consumer is complete, it is best to close it rather than return it to a pool for later reuse: this is because, even if a consumer is idle, ActiveMQ will keep delivering messages to the consumer's prefetch buffer, where they'll get held until the consumer is active again. If you are creating a collection of consumers (for example, for multi-threaded message consumption), you might want to consider using a lower prefetch value for each consumer (e.g. 10 or 20), to ensure that all messages don't end up going to just one of the consumers. See this FAQ entry for more detail: http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html "
Hope this clarifies things.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Dec 8, 2011, at 6:10 AM, Jason Dillon wrote:
> Does the activemq-pool stuff cope with pooling connections, sessions and producers? Such that a component could access create and use these normally + close them, and under the covers activemq-pool will do the smart thing and reuse/avoid-close? Consumers are not pooled in similar fashion?
>
> I believe ^^^ is the case, but I just wanted to confirm incase I'm misunderstanding something.
>
> --jason
>
>
> On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
>
>> Hi Jason,
>>
>> those operations are costly and if your component must open/close it for
>> every message it will affect performances. In those cases it is recommended
>> to use pool connection factory which caches those object and improve
>> performances.
>>
>> See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
>> on this topic (in case of Spring)
>>
>> Regards
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>> -----------------
>> The experts in open source integration and messaging - http://fusesource.com
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
>>
>> On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>>
>>> I'm wondering what sort of overhead there is to create and then close) the
>>> components needed to send a message, specifically after you have a started
>>> connection and using a vm:// transport.
>>>
>>> I'm working on implementing distributed eventing for a server which
>>> already has its own eventing built-in (so adapting its events to JMS
>>> messages published to topics). The events can come from any thread and be
>>> sent to different topics based source event details. That seems to mean
>>> that for each local event I have to:
>>>
>>> 1) reference destination
>>> 2) create session
>>> 3) create producer
>>> 4) build message for event and send
>>> 5 ) close producer and session (discard destination)
>>>
>>> #1 looks like its just object creation, but has some parsing of physical
>>> name (quite a few ops as it looks like)... so could potentially cache these
>>> (trade a bit of memory for a string lookup over always creating new
>>> instance)?
>>>
>>> Not sure what overhead there is for #2, #3 or #5. Is there any
>>> documentation on roughly what these operations cost?
>>>
>>> The destination + session could change so #3 would have to be done
>>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>>> better to have the local event handler queue up the publish in a
>>> BlockingQueue (or similar) so that a single thread + session (or
>>> potentially small pool of thread+session) could be used to a actually
>>> perform the publish?
>>>
>>> Does anyone have any insight on to what would be best option for least
>>> overhead for this use-case?
>>>
>>> --jason
>
Re: Overhead of component creation to send message
Posted by Jason Dillon <ja...@planet57.com>.
Does the activemq-pool stuff cope with pooling connections, sessions and producers? Such that a component could access create and use these normally + close them, and under the covers activemq-pool will do the smart thing and reuse/avoid-close? Consumers are not pooled in similar fashion?
I believe ^^^ is the case, but I just wanted to confirm incase I'm misunderstanding something.
--jason
On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
> Hi Jason,
>
> those operations are costly and if your component must open/close it for
> every message it will affect performances. In those cases it is recommended
> to use pool connection factory which caches those object and improve
> performances.
>
> See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
> on this topic (in case of Spring)
>
> Regards
> --
> Dejan Bosanac - http://twitter.com/dejanb
> -----------------
> The experts in open source integration and messaging - http://fusesource.com
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
> On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>
>> I'm wondering what sort of overhead there is to create and then close) the
>> components needed to send a message, specifically after you have a started
>> connection and using a vm:// transport.
>>
>> I'm working on implementing distributed eventing for a server which
>> already has its own eventing built-in (so adapting its events to JMS
>> messages published to topics). The events can come from any thread and be
>> sent to different topics based source event details. That seems to mean
>> that for each local event I have to:
>>
>> 1) reference destination
>> 2) create session
>> 3) create producer
>> 4) build message for event and send
>> 5 ) close producer and session (discard destination)
>>
>> #1 looks like its just object creation, but has some parsing of physical
>> name (quite a few ops as it looks like)... so could potentially cache these
>> (trade a bit of memory for a string lookup over always creating new
>> instance)?
>>
>> Not sure what overhead there is for #2, #3 or #5. Is there any
>> documentation on roughly what these operations cost?
>>
>> The destination + session could change so #3 would have to be done
>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>> better to have the local event handler queue up the publish in a
>> BlockingQueue (or similar) so that a single thread + session (or
>> potentially small pool of thread+session) could be used to a actually
>> perform the publish?
>>
>> Does anyone have any insight on to what would be best option for least
>> overhead for this use-case?
>>
>> --jason
Re: Overhead of component creation to send message
Posted by Dejan Bosanac <de...@nighttale.net>.
Sure, take a look at
http://camel.apache.org/activemq.html
especially the part on connection pooling, which essential for efficient
usage of resources.
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Nov 22, 2011 at 8:40 PM, Jason Dillon <ja...@planet57.com> wrote:
> Is there any Camel component which could be used here to perform the hand
> off of an event to a thread to encode into message and publish to a topic
> which would perform optimally?
>
> --jason
>
>
> On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
>
> > Hi Jason,
> >
> > those operations are costly and if your component must open/close it for
> > every message it will affect performances. In those cases it is
> recommended
> > to use pool connection factory which caches those object and improve
> > performances.
> >
> > See http://activemq.apache.org/jmstemplate-gotchas.html for some more
> info
> > on this topic (in case of Spring)
> >
> > Regards
> > --
> > Dejan Bosanac - http://twitter.com/dejanb
> > -----------------
> > The experts in open source integration and messaging -
> http://fusesource.com
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
> >
> > On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com>
> wrote:
> >
> >> I'm wondering what sort of overhead there is to create and then close)
> the
> >> components needed to send a message, specifically after you have a
> started
> >> connection and using a vm:// transport.
> >>
> >> I'm working on implementing distributed eventing for a server which
> >> already has its own eventing built-in (so adapting its events to JMS
> >> messages published to topics). The events can come from any thread and
> be
> >> sent to different topics based source event details. That seems to mean
> >> that for each local event I have to:
> >>
> >> 1) reference destination
> >> 2) create session
> >> 3) create producer
> >> 4) build message for event and send
> >> 5 ) close producer and session (discard destination)
> >>
> >> #1 looks like its just object creation, but has some parsing of physical
> >> name (quite a few ops as it looks like)... so could potentially cache
> these
> >> (trade a bit of memory for a string lookup over always creating new
> >> instance)?
> >>
> >> Not sure what overhead there is for #2, #3 or #5. Is there any
> >> documentation on roughly what these operations cost?
> >>
> >> The destination + session could change so #3 would have to be done
> >> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps
> its
> >> better to have the local event handler queue up the publish in a
> >> BlockingQueue (or similar) so that a single thread + session (or
> >> potentially small pool of thread+session) could be used to a actually
> >> perform the publish?
> >>
> >> Does anyone have any insight on to what would be best option for least
> >> overhead for this use-case?
> >>
> >> --jason
>
>
Re: Overhead of component creation to send message
Posted by Torsten Mielke <to...@fusesource.com>.
The camel-jms component can be used to consume and produce msgs. But Camel itself does not pool or cache anything. It depends on the underlying Spring DefaultMessageListenerContainer and connection factory to provide pooling.
The Camel installation has a jms example in examples/camel-example-jms-file that can serve as a starting point.
Hope this helps.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Nov 22, 2011, at 8:40 PM, Jason Dillon wrote:
> Is there any Camel component which could be used here to perform the hand off of an event to a thread to encode into message and publish to a topic which would perform optimally?
>
> --jason
>
>
> On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
>
>> Hi Jason,
>>
>> those operations are costly and if your component must open/close it for
>> every message it will affect performances. In those cases it is recommended
>> to use pool connection factory which caches those object and improve
>> performances.
>>
>> See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
>> on this topic (in case of Spring)
>>
>> Regards
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>> -----------------
>> The experts in open source integration and messaging - http://fusesource.com
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
>>
>> On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>>
>>> I'm wondering what sort of overhead there is to create and then close) the
>>> components needed to send a message, specifically after you have a started
>>> connection and using a vm:// transport.
>>>
>>> I'm working on implementing distributed eventing for a server which
>>> already has its own eventing built-in (so adapting its events to JMS
>>> messages published to topics). The events can come from any thread and be
>>> sent to different topics based source event details. That seems to mean
>>> that for each local event I have to:
>>>
>>> 1) reference destination
>>> 2) create session
>>> 3) create producer
>>> 4) build message for event and send
>>> 5 ) close producer and session (discard destination)
>>>
>>> #1 looks like its just object creation, but has some parsing of physical
>>> name (quite a few ops as it looks like)... so could potentially cache these
>>> (trade a bit of memory for a string lookup over always creating new
>>> instance)?
>>>
>>> Not sure what overhead there is for #2, #3 or #5. Is there any
>>> documentation on roughly what these operations cost?
>>>
>>> The destination + session could change so #3 would have to be done
>>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>>> better to have the local event handler queue up the publish in a
>>> BlockingQueue (or similar) so that a single thread + session (or
>>> potentially small pool of thread+session) could be used to a actually
>>> perform the publish?
>>>
>>> Does anyone have any insight on to what would be best option for least
>>> overhead for this use-case?
>>>
>>> --jason
>
Re: Overhead of component creation to send message
Posted by Jason Dillon <ja...@planet57.com>.
Is there any Camel component which could be used here to perform the hand off of an event to a thread to encode into message and publish to a topic which would perform optimally?
--jason
On Nov 18, 2011, at 2:27 AM, Dejan Bosanac wrote:
> Hi Jason,
>
> those operations are costly and if your component must open/close it for
> every message it will affect performances. In those cases it is recommended
> to use pool connection factory which caches those object and improve
> performances.
>
> See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
> on this topic (in case of Spring)
>
> Regards
> --
> Dejan Bosanac - http://twitter.com/dejanb
> -----------------
> The experts in open source integration and messaging - http://fusesource.com
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
> On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>
>> I'm wondering what sort of overhead there is to create and then close) the
>> components needed to send a message, specifically after you have a started
>> connection and using a vm:// transport.
>>
>> I'm working on implementing distributed eventing for a server which
>> already has its own eventing built-in (so adapting its events to JMS
>> messages published to topics). The events can come from any thread and be
>> sent to different topics based source event details. That seems to mean
>> that for each local event I have to:
>>
>> 1) reference destination
>> 2) create session
>> 3) create producer
>> 4) build message for event and send
>> 5 ) close producer and session (discard destination)
>>
>> #1 looks like its just object creation, but has some parsing of physical
>> name (quite a few ops as it looks like)... so could potentially cache these
>> (trade a bit of memory for a string lookup over always creating new
>> instance)?
>>
>> Not sure what overhead there is for #2, #3 or #5. Is there any
>> documentation on roughly what these operations cost?
>>
>> The destination + session could change so #3 would have to be done
>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>> better to have the local event handler queue up the publish in a
>> BlockingQueue (or similar) so that a single thread + session (or
>> potentially small pool of thread+session) could be used to a actually
>> perform the publish?
>>
>> Does anyone have any insight on to what would be best option for least
>> overhead for this use-case?
>>
>> --jason
Reply:Re: Overhead of component creation to send message
Posted by SuoNayi <su...@163.com>.
With JmsTemplate, it's suffering to create and destory the underlying resource with any call on it, so from Spring 2.5.3,CachingConnectionFactory is introduced with new features such as caching connection,session,consumer and producer resouce.So donot be afraid of JmsTemplate anymore.
At 2011-11-18 18:27:40,"Dejan Bosanac" <de...@nighttale.net> wrote:
>Hi Jason,
>
>those operations are costly and if your component must open/close it for
>every message it will affect performances. In those cases it is recommended
>to use pool connection factory which caches those object and improve
>performances.
>
>See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
>on this topic (in case of Spring)
>
>Regards
>--
>Dejan Bosanac - http://twitter.com/dejanb
>-----------------
>The experts in open source integration and messaging - http://fusesource.com
>ActiveMQ in Action - http://www.manning.com/snyder/
>Blog - http://www.nighttale.net
>
>
>On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
>
>> I'm wondering what sort of overhead there is to create and then close) the
>> components needed to send a message, specifically after you have a started
>> connection and using a vm:// transport.
>>
>> I'm working on implementing distributed eventing for a server which
>> already has its own eventing built-in (so adapting its events to JMS
>> messages published to topics). The events can come from any thread and be
>> sent to different topics based source event details. That seems to mean
>> that for each local event I have to:
>>
>> 1) reference destination
>> 2) create session
>> 3) create producer
>> 4) build message for event and send
>> 5 ) close producer and session (discard destination)
>>
>> #1 looks like its just object creation, but has some parsing of physical
>> name (quite a few ops as it looks like)... so could potentially cache these
>> (trade a bit of memory for a string lookup over always creating new
>> instance)?
>>
>> Not sure what overhead there is for #2, #3 or #5. Is there any
>> documentation on roughly what these operations cost?
>>
>> The destination + session could change so #3 would have to be done
>> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
>> better to have the local event handler queue up the publish in a
>> BlockingQueue (or similar) so that a single thread + session (or
>> potentially small pool of thread+session) could be used to a actually
>> perform the publish?
>>
>> Does anyone have any insight on to what would be best option for least
>> overhead for this use-case?
>>
>> --jason
Re: Overhead of component creation to send message
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi Jason,
those operations are costly and if your component must open/close it for
every message it will affect performances. In those cases it is recommended
to use pool connection factory which caches those object and improve
performances.
See http://activemq.apache.org/jmstemplate-gotchas.html for some more info
on this topic (in case of Spring)
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Nov 18, 2011 at 1:30 AM, Jason Dillon <ja...@planet57.com> wrote:
> I'm wondering what sort of overhead there is to create and then close) the
> components needed to send a message, specifically after you have a started
> connection and using a vm:// transport.
>
> I'm working on implementing distributed eventing for a server which
> already has its own eventing built-in (so adapting its events to JMS
> messages published to topics). The events can come from any thread and be
> sent to different topics based source event details. That seems to mean
> that for each local event I have to:
>
> 1) reference destination
> 2) create session
> 3) create producer
> 4) build message for event and send
> 5 ) close producer and session (discard destination)
>
> #1 looks like its just object creation, but has some parsing of physical
> name (quite a few ops as it looks like)... so could potentially cache these
> (trade a bit of memory for a string lookup over always creating new
> instance)?
>
> Not sure what overhead there is for #2, #3 or #5. Is there any
> documentation on roughly what these operations cost?
>
> The destination + session could change so #3 would have to be done
> anyways, hopefully its cheap? If #2 is not super cheap, then perhaps its
> better to have the local event handler queue up the publish in a
> BlockingQueue (or similar) so that a single thread + session (or
> potentially small pool of thread+session) could be used to a actually
> perform the publish?
>
> Does anyone have any insight on to what would be best option for least
> overhead for this use-case?
>
> --jason