You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ohad <oh...@eng.tau.ac.il> on 2006/07/16 15:38:04 UTC
activemq clients all hang
Hi All
I am developing a high load multi-process application which is sending a
fairly large amount of messages . I am using version 4.0.1 and those
features :
1.Connections are standard, non durable, non auto-reconnect.
2.sessions are non transactional , AutoAcknowledge
3.temporary queues are created one for each connection , and usually a
process has one
4.JMSConnection instance and multiple session objects.
5. I am using the ActiveMQConnection.destroyDestination(...) function
sometimes to destroy destinations that won't be used any more
The processes work fine for 3-4 hours , and then , i experience those
symptoms :
1. every MessageProducer.send() function hangs and doesn't return
2. Queues are not being emptied : i have a code that browses the queue ,
find the number of messages out of it , and the makes a receive() function
for every message. The queue is not empty even after the messages are
consumed.
3. memory consumption is normal for all i know.
I haven't compiled a juint test to recreate it , but it may be the only way
to handle this problem...
Any idea anyone ?
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5348918
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by James Strachan <ja...@gmail.com>.
On 7/23/06, ohad <oh...@eng.tau.ac.il> wrote:
>
> This problem is back again...
> I thought i solved it but the minute i have put some more load on the system
> it hanged all the processes. It seems that there is some deadlock that
> locaks all the send() and createConsumer() calls - and i can't reconstruct
> the problem.
If its really a deadlock then you should be able to create a stack
trace and show us the objects which are deadlocking.
--
James
-------
http://radio.weblogs.com/0112098/
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
Do we have any ETA for this. I know I just submitted this yesterday, but
this is become really critical for us. Right now, we are grinding to halt
every day, and we cannot work that way.
Chris
Christopher Mihaly wrote:
>
> Ok, I submitted an issue, and I included a junit test case that in my case
> is 100% repeatatble. This one is a big problem for me, because this
> combined with another problem I have with off-line subscriptions getting
> stuck, I have to kill my topic after a few days which breaks a lot of
> stuff.
>
> Thanks
> Chris
>
>
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5508016
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
Ok, I submitted an issue, and I included a junit test case that in my case is
100% repeatatble. This one is a big problem for me, because this combined
with another problem I have with off-line subscriptions getting stuck, I
have to kill my topic after a few days which breaks a lot of stuff.
Thanks
Chris
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5488930
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by James Strachan <ja...@gmail.com>.
Could you raise a JIRA for this - its sounding like the cache eviction
for durable topics is not kicking in and things are blocking on the
amount of RAM in the <usageManager/>
On 7/25/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>
> There are no non-durable consumers. It is a pretty simple test, I created
> a durable subscriber that subscribed for messages on the topic. Then I shut
> down that consumer and wrote a publisher that wrote persistent messages, up
> to 100,000 of them. It will have after about 1800 messages every time. If
> I then run my druable consumer, it will read the messages off the queue and
> I can send another few messages before it hangs.
>
> If I leave the durable subscriber inactive for a while, it will go OFF-LINE
> and sometimes I am unable to get it to back on-lne ever. But that is a
> different message thread.
>
> Chris
>
>
>
> James.Strachan wrote:
> >
> > Do you have any slow consumers (non-durable consumers) who are not keeping
> > up?
> >
> > On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
> >>
> >>
> >> James.Strachan wrote:
> >> >
> >> > Are you sending the messages with persistent message delivery mode?
> >> > private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
> >> > private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
> >> > publisher.send(msg, deliveryMode, deliveryPriority,
> >> > Message.DEFAULT_TIME_TO_LIVE);
> >> > Also what version are you using?
> >> >
> >> > Version Version 4.0.1
> >> >
> >> > On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
> >> >>
> >> >> On the test case, I created a topic publisher and create 100000
> >> >> messages
> >> >> using asyncsend. This works great. Then I make a durable
> >> subscription
> >> >> and
> >> >> that client exits leaving his durable subsription. Then I send the
> >> >> publisher and around 2000 messages it will hang. The only solution
> >> to
> >> >> this is to run my client that subscribed to the subscription and read
> >> the
> >> >> thousands of messages. Then I can send another couple thousand
> >> >> messages,
> >> >> etc. It is completely repeatable. This happens if I am using
> >> >> journaling
> >> >> or not, or out of the box, derby or MS SQLServer as the persistenace
> >> DB.
> >> >> I believe this is what is happening on our production server. We
> >> have
> >> >> clients that are run weekly, or at other unspecified times that will
> >> hold
> >> >> durable subscriptions to the topics. I have not found any way around
> >> >> this
> >> >> other than killing off subscriptions (sometimes not possible since you
> >> >> cannot destroy a subsription that is off-line, not sure why they are
> >> >> off-line, but I guess over time they become off-line). So the only
> >> >> solutoins is to destroy the topic and recreate, but this destroys a
> >> lot
> >> >> of
> >> >> data for us and is getting unusable :_(
> >> >>
> >> >> Chris
> >> >>
> >> >>
> >> >> Chris
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
> >> >> Sent from the ActiveMQ - User forum at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> >
> >> > James
> >> > -------
> >> > http://radio.weblogs.com/0112098/
> >> >
> >> >
> >> --
> >> View this message in context:
> >> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
> >> Sent from the ActiveMQ - User forum at Nabble.com.
> >>
> >>
> >
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5480407
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
There are no non-durable consumers. It is a pretty simple test, I created
a durable subscriber that subscribed for messages on the topic. Then I shut
down that consumer and wrote a publisher that wrote persistent messages, up
to 100,000 of them. It will have after about 1800 messages every time. If
I then run my druable consumer, it will read the messages off the queue and
I can send another few messages before it hangs.
If I leave the durable subscriber inactive for a while, it will go OFF-LINE
and sometimes I am unable to get it to back on-lne ever. But that is a
different message thread.
Chris
James.Strachan wrote:
>
> Do you have any slow consumers (non-durable consumers) who are not keeping
> up?
>
> On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>>
>>
>> James.Strachan wrote:
>> >
>> > Are you sending the messages with persistent message delivery mode?
>> > private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
>> > private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
>> > publisher.send(msg, deliveryMode, deliveryPriority,
>> > Message.DEFAULT_TIME_TO_LIVE);
>> > Also what version are you using?
>> >
>> > Version Version 4.0.1
>> >
>> > On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>> >>
>> >> On the test case, I created a topic publisher and create 100000
>> >> messages
>> >> using asyncsend. This works great. Then I make a durable
>> subscription
>> >> and
>> >> that client exits leaving his durable subsription. Then I send the
>> >> publisher and around 2000 messages it will hang. The only solution
>> to
>> >> this is to run my client that subscribed to the subscription and read
>> the
>> >> thousands of messages. Then I can send another couple thousand
>> >> messages,
>> >> etc. It is completely repeatable. This happens if I am using
>> >> journaling
>> >> or not, or out of the box, derby or MS SQLServer as the persistenace
>> DB.
>> >> I believe this is what is happening on our production server. We
>> have
>> >> clients that are run weekly, or at other unspecified times that will
>> hold
>> >> durable subscriptions to the topics. I have not found any way around
>> >> this
>> >> other than killing off subscriptions (sometimes not possible since you
>> >> cannot destroy a subsription that is off-line, not sure why they are
>> >> off-line, but I guess over time they become off-line). So the only
>> >> solutoins is to destroy the topic and recreate, but this destroys a
>> lot
>> >> of
>> >> data for us and is getting unusable :_(
>> >>
>> >> Chris
>> >>
>> >>
>> >> Chris
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
>> >> Sent from the ActiveMQ - User forum at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> >
>> > James
>> > -------
>> > http://radio.weblogs.com/0112098/
>> >
>> >
>> --
>> View this message in context:
>> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5480407
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by James Strachan <ja...@gmail.com>.
Do you have any slow consumers (non-durable consumers) who are not keeping up?
On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>
>
> James.Strachan wrote:
> >
> > Are you sending the messages with persistent message delivery mode?
> > private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
> > private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
> > publisher.send(msg, deliveryMode, deliveryPriority,
> > Message.DEFAULT_TIME_TO_LIVE);
> > Also what version are you using?
> >
> > Version Version 4.0.1
> >
> > On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
> >>
> >> On the test case, I created a topic publisher and create 100000
> >> messages
> >> using asyncsend. This works great. Then I make a durable subscription
> >> and
> >> that client exits leaving his durable subsription. Then I send the
> >> publisher and around 2000 messages it will hang. The only solution to
> >> this is to run my client that subscribed to the subscription and read the
> >> thousands of messages. Then I can send another couple thousand
> >> messages,
> >> etc. It is completely repeatable. This happens if I am using
> >> journaling
> >> or not, or out of the box, derby or MS SQLServer as the persistenace DB.
> >> I believe this is what is happening on our production server. We have
> >> clients that are run weekly, or at other unspecified times that will hold
> >> durable subscriptions to the topics. I have not found any way around
> >> this
> >> other than killing off subscriptions (sometimes not possible since you
> >> cannot destroy a subsription that is off-line, not sure why they are
> >> off-line, but I guess over time they become off-line). So the only
> >> solutoins is to destroy the topic and recreate, but this destroys a lot
> >> of
> >> data for us and is getting unusable :_(
> >>
> >> Chris
> >>
> >>
> >> Chris
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
> >> Sent from the ActiveMQ - User forum at Nabble.com.
> >>
> >>
> >
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
> --
> View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
James.Strachan wrote:
>
> Are you sending the messages with persistent message delivery mode?
> private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
> private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
> publisher.send(msg, deliveryMode, deliveryPriority,
> Message.DEFAULT_TIME_TO_LIVE);
> Also what version are you using?
>
> Version Version 4.0.1
>
> On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>>
>> On the test case, I created a topic publisher and create 100000
>> messages
>> using asyncsend. This works great. Then I make a durable subscription
>> and
>> that client exits leaving his durable subsription. Then I send the
>> publisher and around 2000 messages it will hang. The only solution to
>> this is to run my client that subscribed to the subscription and read the
>> thousands of messages. Then I can send another couple thousand
>> messages,
>> etc. It is completely repeatable. This happens if I am using
>> journaling
>> or not, or out of the box, derby or MS SQLServer as the persistenace DB.
>> I believe this is what is happening on our production server. We have
>> clients that are run weekly, or at other unspecified times that will hold
>> durable subscriptions to the topics. I have not found any way around
>> this
>> other than killing off subscriptions (sometimes not possible since you
>> cannot destroy a subsription that is off-line, not sure why they are
>> off-line, but I guess over time they become off-line). So the only
>> solutoins is to destroy the topic and recreate, but this destroys a lot
>> of
>> data for us and is getting unusable :_(
>>
>> Chris
>>
>>
>> Chris
>>
>> --
>> View this message in context:
>> http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by James Strachan <ja...@gmail.com>.
Are you sending the messages with persistent message delivery mode?
Also what version are you using?
On 7/24/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>
> On the test case, I created a topic publisher and create 100000 messages
> using asyncsend. This works great. Then I make a durable subscription and
> that client exits leaving his durable subsription. Then I send the
> publisher and around 2000 messages it will hang. The only solution to
> this is to run my client that subscribed to the subscription and read the
> thousands of messages. Then I can send another couple thousand messages,
> etc. It is completely repeatable. This happens if I am using journaling
> or not, or out of the box, derby or MS SQLServer as the persistenace DB.
> I believe this is what is happening on our production server. We have
> clients that are run weekly, or at other unspecified times that will hold
> durable subscriptions to the topics. I have not found any way around this
> other than killing off subscriptions (sometimes not possible since you
> cannot destroy a subsription that is off-line, not sure why they are
> off-line, but I guess over time they become off-line). So the only
> solutoins is to destroy the topic and recreate, but this destroys a lot of
> data for us and is getting unusable :_(
>
> Chris
>
>
> Chris
>
> --
> View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable subscription and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution to
this is to run my client that subscribed to the subscription and read the
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace DB.
I believe this is what is happening on our production server. We have
clients that are run weekly, or at other unspecified times that will hold
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a lot of
data for us and is getting unusable :_(
Chris
Chris
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by ohad <oh...@eng.tau.ac.il>.
This problem is back again...
I thought i solved it but the minute i have put some more load on the system
it hanged all the processes. It seems that there is some deadlock that
locaks all the send() and createConsumer() calls - and i can't reconstruct
the problem.
Christpher , can you describe what you are doing exactly so we can somehow
see if we have some things in common ?
Anyone ?
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5453634
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by ohad <oh...@eng.tau.ac.il>.
I have managed to find a peace of code that hangs my clients.
Here is the situation :
1. a process starts , opening connections and a consumer
2. some producers send messages to the queue
3. the process terminates abruptly without closing the connection or the
consumer
4. when i try and open a consumer on the same queue - it hangs.
I have some code here that gets activemq stuck. I hope someone will try it
out :
(Please note that you have to run it twice in order to have it stuck...)
package dimes.taskloader.testing.manual;
import java.io.IOException;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;
/**
* @author Ohad Serfaty
*
*/
public class HangTest {
public void testMe() throws IOException, InterruptedException,
JMSException{
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
ActiveMQConnection connection = (ActiveMQConnection)
factory.createConnection();
connection.setAlwaysSessionAsync(true);
final Session session = connection.createSession(false ,
Session.AUTO_ACKNOWLEDGE);
final Queue queue = session.createQueue("test.queue.1");
System.out.println("Creating consumer...");
MessageConsumer consumer = connection.createSession(true ,
Session.AUTO_ACKNOWLEDGE).createConsumer(queue);
System.out.println("Setting message listener...");
consumer.setMessageListener( new MessageListener()
{
public void onMessage(Message arg0) {
System.out.println("Hey!");
}
});
Thread[] pool = new Thread[5];
final MessageProducer[] producers = new MessageProducer[5];
for (int j=0; j<5; j++)
{
producers[j] = session.createProducer(queue);
final int k = j;
pool[j] = new Thread(){
public void run(){
System.out.println("running ?");
try
{
for (int i = 0; i < 1000; i++)
{
producers[k].send(session.createTextMessage("Hello!"));
System.out.println(i + ":Got here ?");
}
}
catch (Exception e)
{
e.printStackTrace();
}
System.out.println("Ended ?");
}
};
}
System.out.println("starting threads...");
for (int i=0; i<5; i++)
pool[i].start();
Thread.sleep(2000);
for (int i=0; i<5; i++)
producers[i].close();
//consumer.close();
session.close();
//connection.close();
}
// Please note that this method has to be run twice ( from two different
processes...)
public static void main(String[] args) throws IOException,
InterruptedException, JMSException
{
HangTest tester = new HangTest();
tester.testMe();
}
}
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5454005
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by ohad <oh...@eng.tau.ac.il>.
here is what i did to reduce the problems ( i have been running my app for
hours now and it seems fine ) :
1. use JMX with jconsole to hook onto activemq , and make sure that every
queue that you opened is closed . it is better to close a MessageProducer
before a MessageConsumer , because then the queue is purged entirely and it
is out of the system.
2. make sure that there are no consumers and producers that are hanging open
- i think that this is the main reason why it happened to me
3. I used ActiveMQConnection.destroyDestination(...) on each of my queues
once they are no longer needed - perhaps you can use it too.
Sorry for having only tips - i can't tell what truly caused that problem...
Ohad Serfaty
DIMES team
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5383368
Sent from the ActiveMQ - User forum at Nabble.com.
Re: activemq clients all hang
Posted by Christopher Mihaly <cm...@hotmail.com>.
I am seeing exactly the same problem. Right now I am completely stuck, every
call to the activemq hangs. There is no diagnostics of note, other than the
subsequent calls all hang witha client already connected because the
original is hung. Any ideas on how to fixt this?
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5367905
Sent from the ActiveMQ - User forum at Nabble.com.