You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Dom <th...@gmail.com> on 2011/09/13 19:18:38 UTC
ActiveMQ Session Task threads disappearing
Hi,
I've starting having unusual problems with my ActiveMQ consumers, we send
object messages via JMS 24 hours a day, peaking at around 10k per second to
various different consumers. All these consumers are consuming from the same
topic, however from different physical servers. The consumers are all based
on the same abstract class (from a shared JAR).
On our servers running Java 6 build 20 we haven't seen an issue. We have
some newer servers running in Java 6 build 26, and these servers some times
have the issue.
The problem is; Some of the consumers will randomly simply stop consuming -
still be sent messages from ActiveMQ, just stop passing them into our code
base. After a while this causes all consumers on that topic to stop
receiving messages (once our buffer on ActiveMQ fills up).
It seems (according to thread dumps) that when this happens we have "lost"
the ActiveMQ session task threads. The transport threads will still be
there, just not the session tasks. We have try catch (Exception) in all our
onMessage methods which *should* catch everything.
I have to restart the jvm to get it to work again, which is not ideal in a
production environment.
Does anybody have any ideas?
Thanks,
Dom
PS: This is our consumer create code;
ActiveMQConnectionFactory connectionFactory = new
ActiveMQConnectionFactory(jmsConfig.getUserName(), jmsConfig.getPassword(),
jmsConfig.getUrl());
connection = connectionFactory.createConnection();
if (clientID.length() > 0) {
connection.setClientID(clientID);
}
connection.setExceptionListener(this);
connection.start();
// transacted is false and session is set to auto ack mode.
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// our destination topic subject on which we receive objects.
Destination destination = session.createTopic(topicSubject);
consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3810649.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Session Task threads disappearing
Posted by Gary Tully <ga...@gmail.com>.
set the uncaught handler once in your client jvm, so that it has global effect:
>From your client main, or some startup code, use:
http://download.oracle.com/javase/1.5.0/docs/api/java/lang/Thread.html#setDefaultUncaughtExceptionHandler(java.lang.Thread.UncaughtExceptionHandler)
On 15 September 2011 14:47, Dom <th...@gmail.com> wrote:
> I'm hesitant to turn trace logging on as we get thousands of messages per
> second, and this problem occurs maybe weeks apart, maybe days apart and so
> trying to find anything in the log will be hard.
>
> As for the default uncaught exception handler... I cant see how we can set
> that from outside the ActiveMQ library. And our onMessage code is always;
>
> public void onMessage(Message message) {
> try {
> //handling code here
> } catch (Exception e) {
> log.error("Exception parsing message from JMS", e);
> }
> }
>
> From my consumer I've added a line to always set the
> uncaughtExceptionHandler in our onMessage and set to log that.
>
> I've also noticed from one of the JVM's that it's been up for; 5 days
> 23:30:52.042
> and the current active session task is; ActiveMQ Session Task-7296
>
> I'm not sure if that's any help, because from what I understand of the
> thread pool executors,they will unload threads if idle and create more if
> needed.
>
>
> On Thu, Sep 15, 2011 at 12:31 PM, Gary Tully [via ActiveMQ] <
> ml-node+s2283324n3815303h42@n4.nabble.com> wrote:
>
>> Dom,
>> this is unusual, some thoughts:
>> can you enable trace level logging for activemq on your clients to
>> see if there is anything suspicious in there.
>> If the dispatch thread is dying we should be able to see why.
>> As a catch all, can you specify a
>> java.lang.Thread#setDefaultUncaughtExceptionHandler in your client
>> code, in case some catch is missing.
>>
>> The other thing we could do it check that the dispatch thread is alive
>> on each message receive from the transport, but really we should be
>> getting some indication as to why it dies.
>>
>> On 14 September 2011 15:45, Dom <[hidden email]<http://user/SendEmail.jtp?type=node&node=3815303&i=0>>
>> wrote:
>>
>> > Our subscribers are not durable, however I have noticed that it does
>> appear
>> > to occur around our peaks (sometimes before, sometimes afterwards).
>> Although
>> > I have had it occur once at night, when our traffic was very load (1% of
>> > peak).
>> >
>> > I'm using the OpenWire protocol, not stomp. So I'm guessing that our two
>> > problems may be different. As I said before, I've noticed that the thread
>>
>> > which deals with handing messages from the transport code to
>> > my listeners seems to die, and even killing and restart the consumer
>> doesn't
>> > work (although it does free up the topic for the other consumers). I have
>> to
>> > actively kill the client's JVM and restart it.
>> >
>> > Dom
>> >
>> > On Wed, Sep 14, 2011 at 1:14 PM, Aleksandar Ivanisevic-3 [via ActiveMQ] <
>>
>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3815303&i=1>>
>> wrote:
>> >
>> >> Dom <[hidden email] <
>> http://user/SendEmail.jtp?type=node&node=3812638&i=0>>
>> >> writes:
>> >>
>> >> > Hi,
>> >> >
>> >> > I've starting having unusual problems with my ActiveMQ consumers, we
>> send
>> >>
>> >> > object messages via JMS 24 hours a day, peaking at around 10k per
>> second
>> >> to
>> >> > various different consumers. All these consumers are consuming from
>> the
>> >> same
>> >> > topic, however from different physical servers. The consumers are all
>> >> based
>> >> > on the same abstract class (from a shared JAR).
>> >> >
>> >> > On our servers running Java 6 build 20 we haven't seen an issue. We
>> have
>> >> > some newer servers running in Java 6 build 26, and these servers some
>> >> times
>> >> > have the issue.
>> >> >
>> >> > The problem is; Some of the consumers will randomly simply stop
>> consuming
>> >> -
>> >> > still be sent messages from ActiveMQ, just stop passing them into our
>> >> code
>> >> > base. After a while this causes all consumers on that topic to stop
>> >> > receiving messages (once our buffer on ActiveMQ fills up).
>> >>
>> >> Are you using durable subscribers? Have you seen redeliveries around
>> >> the time it stops? I'm having a similar issue where a consumer just
>> >> stops receiving messages from a durable topic, usually after a period
>> >> of peak traffic and after a redelivery. After a while this causes the
>> >> complete topic to "block". Consumers are subscribed to multiple topics
>> >> and only one topic is dead, the rest continue to work normally until a
>> >> consumer is restarted.
>> >>
>> >> Out of desperation I've upgraded the broker to a 5.6 snapshot, and now
>> >> I see this issue a lot less, but still see it once a week on
>> >> average. Haven't been able to tie it to java version, was happening
>> >> with build 20 and now with 23. Consumers are not java at all, they are
>> >> in perl and use STOMP to communicate.
>> >>
>> >>
>> >> [...]
>> >>
>> >>
>> >> --
>> >> Ti si arogantan, prepotentan i peglaš vlastitu frustraciju. -- Ivan
>> >> Tišljar, hr.comp.os.linux
>> >>
>> >>
>> >>
>> >> ------------------------------
>> >> If you reply to this email, your message will be added to the
>> discussion
>> >> below:
>> >>
>> >>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3812638.html
>> >> To unsubscribe from ActiveMQ Session Task threads disappearing, click
>> >> here<
>>
>> >>
>> >>
>> >
>> >
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3813089.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3815303.html
>> To unsubscribe from ActiveMQ Session Task threads disappearing, click
>> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3810649&code=dGhlcGhlbml4QGdtYWlsLmNvbXwzODEwNjQ5fC00OTkwNjE4Nzg=>.
>>
>>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3815617.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: ActiveMQ Session Task threads disappearing
Posted by Dom <th...@gmail.com>.
I'm hesitant to turn trace logging on as we get thousands of messages per
second, and this problem occurs maybe weeks apart, maybe days apart and so
trying to find anything in the log will be hard.
As for the default uncaught exception handler... I cant see how we can set
that from outside the ActiveMQ library. And our onMessage code is always;
public void onMessage(Message message) {
try {
//handling code here
} catch (Exception e) {
log.error("Exception parsing message from JMS", e);
}
}
>From my consumer I've added a line to always set the
uncaughtExceptionHandler in our onMessage and set to log that.
I've also noticed from one of the JVM's that it's been up for; 5 days
23:30:52.042
and the current active session task is; ActiveMQ Session Task-7296
I'm not sure if that's any help, because from what I understand of the
thread pool executors,they will unload threads if idle and create more if
needed.
On Thu, Sep 15, 2011 at 12:31 PM, Gary Tully [via ActiveMQ] <
ml-node+s2283324n3815303h42@n4.nabble.com> wrote:
> Dom,
> this is unusual, some thoughts:
> can you enable trace level logging for activemq on your clients to
> see if there is anything suspicious in there.
> If the dispatch thread is dying we should be able to see why.
> As a catch all, can you specify a
> java.lang.Thread#setDefaultUncaughtExceptionHandler in your client
> code, in case some catch is missing.
>
> The other thing we could do it check that the dispatch thread is alive
> on each message receive from the transport, but really we should be
> getting some indication as to why it dies.
>
> On 14 September 2011 15:45, Dom <[hidden email]<http://user/SendEmail.jtp?type=node&node=3815303&i=0>>
> wrote:
>
> > Our subscribers are not durable, however I have noticed that it does
> appear
> > to occur around our peaks (sometimes before, sometimes afterwards).
> Although
> > I have had it occur once at night, when our traffic was very load (1% of
> > peak).
> >
> > I'm using the OpenWire protocol, not stomp. So I'm guessing that our two
> > problems may be different. As I said before, I've noticed that the thread
>
> > which deals with handing messages from the transport code to
> > my listeners seems to die, and even killing and restart the consumer
> doesn't
> > work (although it does free up the topic for the other consumers). I have
> to
> > actively kill the client's JVM and restart it.
> >
> > Dom
> >
> > On Wed, Sep 14, 2011 at 1:14 PM, Aleksandar Ivanisevic-3 [via ActiveMQ] <
>
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3815303&i=1>>
> wrote:
> >
> >> Dom <[hidden email] <
> http://user/SendEmail.jtp?type=node&node=3812638&i=0>>
> >> writes:
> >>
> >> > Hi,
> >> >
> >> > I've starting having unusual problems with my ActiveMQ consumers, we
> send
> >>
> >> > object messages via JMS 24 hours a day, peaking at around 10k per
> second
> >> to
> >> > various different consumers. All these consumers are consuming from
> the
> >> same
> >> > topic, however from different physical servers. The consumers are all
> >> based
> >> > on the same abstract class (from a shared JAR).
> >> >
> >> > On our servers running Java 6 build 20 we haven't seen an issue. We
> have
> >> > some newer servers running in Java 6 build 26, and these servers some
> >> times
> >> > have the issue.
> >> >
> >> > The problem is; Some of the consumers will randomly simply stop
> consuming
> >> -
> >> > still be sent messages from ActiveMQ, just stop passing them into our
> >> code
> >> > base. After a while this causes all consumers on that topic to stop
> >> > receiving messages (once our buffer on ActiveMQ fills up).
> >>
> >> Are you using durable subscribers? Have you seen redeliveries around
> >> the time it stops? I'm having a similar issue where a consumer just
> >> stops receiving messages from a durable topic, usually after a period
> >> of peak traffic and after a redelivery. After a while this causes the
> >> complete topic to "block". Consumers are subscribed to multiple topics
> >> and only one topic is dead, the rest continue to work normally until a
> >> consumer is restarted.
> >>
> >> Out of desperation I've upgraded the broker to a 5.6 snapshot, and now
> >> I see this issue a lot less, but still see it once a week on
> >> average. Haven't been able to tie it to java version, was happening
> >> with build 20 and now with 23. Consumers are not java at all, they are
> >> in perl and use STOMP to communicate.
> >>
> >>
> >> [...]
> >>
> >>
> >> --
> >> Ti si arogantan, prepotentan i peglaš vlastitu frustraciju. -- Ivan
> >> Tišljar, hr.comp.os.linux
> >>
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3812638.html
> >> To unsubscribe from ActiveMQ Session Task threads disappearing, click
> >> here<
>
> >>
> >>
> >
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3813089.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3815303.html
> To unsubscribe from ActiveMQ Session Task threads disappearing, click
> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3810649&code=dGhlcGhlbml4QGdtYWlsLmNvbXwzODEwNjQ5fC00OTkwNjE4Nzg=>.
>
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3815617.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Session Task threads disappearing
Posted by Gary Tully <ga...@gmail.com>.
Dom,
this is unusual, some thoughts:
can you enable trace level logging for activemq on your clients to
see if there is anything suspicious in there.
If the dispatch thread is dying we should be able to see why.
As a catch all, can you specify a
java.lang.Thread#setDefaultUncaughtExceptionHandler in your client
code, in case some catch is missing.
The other thing we could do it check that the dispatch thread is alive
on each message receive from the transport, but really we should be
getting some indication as to why it dies.
On 14 September 2011 15:45, Dom <th...@gmail.com> wrote:
> Our subscribers are not durable, however I have noticed that it does appear
> to occur around our peaks (sometimes before, sometimes afterwards). Although
> I have had it occur once at night, when our traffic was very load (1% of
> peak).
>
> I'm using the OpenWire protocol, not stomp. So I'm guessing that our two
> problems may be different. As I said before, I've noticed that the thread
> which deals with handing messages from the transport code to
> my listeners seems to die, and even killing and restart the consumer doesn't
> work (although it does free up the topic for the other consumers). I have to
> actively kill the client's JVM and restart it.
>
> Dom
>
> On Wed, Sep 14, 2011 at 1:14 PM, Aleksandar Ivanisevic-3 [via ActiveMQ] <
> ml-node+s2283324n3812638h27@n4.nabble.com> wrote:
>
>> Dom <[hidden email] <http://user/SendEmail.jtp?type=node&node=3812638&i=0>>
>> writes:
>>
>> > Hi,
>> >
>> > I've starting having unusual problems with my ActiveMQ consumers, we send
>>
>> > object messages via JMS 24 hours a day, peaking at around 10k per second
>> to
>> > various different consumers. All these consumers are consuming from the
>> same
>> > topic, however from different physical servers. The consumers are all
>> based
>> > on the same abstract class (from a shared JAR).
>> >
>> > On our servers running Java 6 build 20 we haven't seen an issue. We have
>> > some newer servers running in Java 6 build 26, and these servers some
>> times
>> > have the issue.
>> >
>> > The problem is; Some of the consumers will randomly simply stop consuming
>> -
>> > still be sent messages from ActiveMQ, just stop passing them into our
>> code
>> > base. After a while this causes all consumers on that topic to stop
>> > receiving messages (once our buffer on ActiveMQ fills up).
>>
>> Are you using durable subscribers? Have you seen redeliveries around
>> the time it stops? I'm having a similar issue where a consumer just
>> stops receiving messages from a durable topic, usually after a period
>> of peak traffic and after a redelivery. After a while this causes the
>> complete topic to "block". Consumers are subscribed to multiple topics
>> and only one topic is dead, the rest continue to work normally until a
>> consumer is restarted.
>>
>> Out of desperation I've upgraded the broker to a 5.6 snapshot, and now
>> I see this issue a lot less, but still see it once a week on
>> average. Haven't been able to tie it to java version, was happening
>> with build 20 and now with 23. Consumers are not java at all, they are
>> in perl and use STOMP to communicate.
>>
>>
>> [...]
>>
>>
>> --
>> Ti si arogantan, prepotentan i peglaš vlastitu frustraciju. -- Ivan
>> Tišljar, hr.comp.os.linux
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3812638.html
>> To unsubscribe from ActiveMQ Session Task threads disappearing, click
>> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3810649&code=dGhlcGhlbml4QGdtYWlsLmNvbXwzODEwNjQ5fC00OTkwNjE4Nzg=>.
>>
>>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3813089.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: ActiveMQ Session Task threads disappearing
Posted by Dom <th...@gmail.com>.
Our subscribers are not durable, however I have noticed that it does appear
to occur around our peaks (sometimes before, sometimes afterwards). Although
I have had it occur once at night, when our traffic was very load (1% of
peak).
I'm using the OpenWire protocol, not stomp. So I'm guessing that our two
problems may be different. As I said before, I've noticed that the thread
which deals with handing messages from the transport code to
my listeners seems to die, and even killing and restart the consumer doesn't
work (although it does free up the topic for the other consumers). I have to
actively kill the client's JVM and restart it.
Dom
On Wed, Sep 14, 2011 at 1:14 PM, Aleksandar Ivanisevic-3 [via ActiveMQ] <
ml-node+s2283324n3812638h27@n4.nabble.com> wrote:
> Dom <[hidden email] <http://user/SendEmail.jtp?type=node&node=3812638&i=0>>
> writes:
>
> > Hi,
> >
> > I've starting having unusual problems with my ActiveMQ consumers, we send
>
> > object messages via JMS 24 hours a day, peaking at around 10k per second
> to
> > various different consumers. All these consumers are consuming from the
> same
> > topic, however from different physical servers. The consumers are all
> based
> > on the same abstract class (from a shared JAR).
> >
> > On our servers running Java 6 build 20 we haven't seen an issue. We have
> > some newer servers running in Java 6 build 26, and these servers some
> times
> > have the issue.
> >
> > The problem is; Some of the consumers will randomly simply stop consuming
> -
> > still be sent messages from ActiveMQ, just stop passing them into our
> code
> > base. After a while this causes all consumers on that topic to stop
> > receiving messages (once our buffer on ActiveMQ fills up).
>
> Are you using durable subscribers? Have you seen redeliveries around
> the time it stops? I'm having a similar issue where a consumer just
> stops receiving messages from a durable topic, usually after a period
> of peak traffic and after a redelivery. After a while this causes the
> complete topic to "block". Consumers are subscribed to multiple topics
> and only one topic is dead, the rest continue to work normally until a
> consumer is restarted.
>
> Out of desperation I've upgraded the broker to a 5.6 snapshot, and now
> I see this issue a lot less, but still see it once a week on
> average. Haven't been able to tie it to java version, was happening
> with build 20 and now with 23. Consumers are not java at all, they are
> in perl and use STOMP to communicate.
>
>
> [...]
>
>
> --
> Ti si arogantan, prepotentan i peglaš vlastitu frustraciju. -- Ivan
> Tišljar, hr.comp.os.linux
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3812638.html
> To unsubscribe from ActiveMQ Session Task threads disappearing, click
> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3810649&code=dGhlcGhlbml4QGdtYWlsLmNvbXwzODEwNjQ5fC00OTkwNjE4Nzg=>.
>
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Session-Task-threads-disappearing-tp3810649p3813089.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Session Task threads disappearing
Posted by Aleksandar Ivanisevic <al...@ivanisevic.de>.
Dom <th...@gmail.com> writes:
> Hi,
>
> I've starting having unusual problems with my ActiveMQ consumers, we send
> object messages via JMS 24 hours a day, peaking at around 10k per second to
> various different consumers. All these consumers are consuming from the same
> topic, however from different physical servers. The consumers are all based
> on the same abstract class (from a shared JAR).
>
> On our servers running Java 6 build 20 we haven't seen an issue. We have
> some newer servers running in Java 6 build 26, and these servers some times
> have the issue.
>
> The problem is; Some of the consumers will randomly simply stop consuming -
> still be sent messages from ActiveMQ, just stop passing them into our code
> base. After a while this causes all consumers on that topic to stop
> receiving messages (once our buffer on ActiveMQ fills up).
Are you using durable subscribers? Have you seen redeliveries around
the time it stops? I'm having a similar issue where a consumer just
stops receiving messages from a durable topic, usually after a period
of peak traffic and after a redelivery. After a while this causes the
complete topic to "block". Consumers are subscribed to multiple topics
and only one topic is dead, the rest continue to work normally until a
consumer is restarted.
Out of desperation I've upgraded the broker to a 5.6 snapshot, and now
I see this issue a lot less, but still see it once a week on
average. Haven't been able to tie it to java version, was happening
with build 20 and now with 23. Consumers are not java at all, they are
in perl and use STOMP to communicate.
[...]
--
Ti si arogantan, prepotentan i peglaš vlastitu frustraciju. -- Ivan
Tišljar, hr.comp.os.linux
Re: ActiveMQ Session Task threads disappearing
Posted by Todd Crone <to...@lexmark.com>.
When I processing a message something might fail that I want it to try again
and after it tries a specified number times, I want to push that message to
another queue for alternate processing. I have looked at JMS retries but
doesn't seem like the way I want to go.
With a valid Message, what is the easiest way to send it to an arbitrary
destinationName after adding a property?
Thanks,
Todd