You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Chirag Pujara <ch...@gmail.com> on 2014/02/05 22:38:27 UTC
Broker Hangs after some time - or does nothing.
I am using Activemq-5.10.SNAPSHOT and implementing with replicated leveldb.
After few messages consumer stop processing messages. Messages just stay on
queue. nothing happens. Client cannot send anymore messges. I dont see any
exception on client side. I dont see any exception on server logs.
I saw some exception in logs :
2014-02-05 14:01:28,247 | DEBUG | Running WriteCheck[tcp://
10.190.225.214:44179] |
org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
InactivityMonitor Worker
2014-02-05 14:01:28,285 | DEBUG | Transport Connection to: tcp://
10.190.237.125:58514 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.237.125:58514@61616
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:724)
2014-02-05 14:01:28,285 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=activemq,connector=clientConnectors,connectorName=openwire,connectionViewType=clientId,connectionName=ID_RHEL6u2-001-1-38845-1391630198629-0_1
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
tcp:///10.190.237.125:58514@61616
2014-02-05 14:01:28,285 | DEBUG | Transport Connection to: tcp://
10.190.237.125:58519 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.237.125:58519@61616
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:724)
2014-02-05 14:01:28,285 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=activemq,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//10.190.237.125_58514
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
tcp:///10.190.237.125:58514@61616
but after that everything in normal.
any suggestions where to start troubleshooting.
thanks,
chirag
Re: Broker Hangs after some time - or does nothing.
Posted by Gary Tully <ga...@gmail.com>.
is there any possibility you could build a test case to capture your use case?
On 27 July 2014 04:24, rking <ra...@shinetech.com> wrote:
> Also using 5.9.1 and our broker hangs almost regularly for the past few
> months. Restarting the broker fixes it, but we cannot have it happening too
> often. Would be good to know if there's any solution to this. Was thinking
> of upgrading to 5.10 but it seems the same problem happens there.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4683707.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://redhat.com
http://blog.garytully.com
Re: Broker Hangs after some time - or does nothing.
Posted by rking <ra...@shinetech.com>.
Also using 5.9.1 and our broker hangs almost regularly for the past few
months. Restarting the broker fixes it, but we cannot have it happening too
often. Would be good to know if there's any solution to this. Was thinking
of upgrading to 5.10 but it seems the same problem happens there.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4683707.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
so far no luck. I see that in our dev environment. i am not sure how to fix
it. if you figure out anything please share.
thanks,
On Fri, May 16, 2014 at 9:15 AM, rgooduru <go...@gmail.com> wrote:
> Are you able to find the cause of this issue, I am facing same issue with
> 5.9.1. We are using 5.9.1 with leveldb, everything worked properly in our
> QA
> but when we migrate to prod we had seen the same behavoir you were
> describing.
>
> I was unable to capture number of connections or heap size etc from prod. I
> am not able to reproduce this in QA or DEV. Is this something to do with
> network connections? Any help would be appreciated.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4681166.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
Posted by rgooduru <go...@gmail.com>.
Are you able to find the cause of this issue, I am facing same issue with
5.9.1. We are using 5.9.1 with leveldb, everything worked properly in our QA
but when we migrate to prod we had seen the same behavoir you were
describing.
I was unable to capture number of connections or heap size etc from prod. I
am not able to reproduce this in QA or DEV. Is this something to do with
network connections? Any help would be appreciated.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4681166.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
I tried again and this time I didnt send bunch of messages at same time. I
noticed that some message gets processed right away but some messges stays
on queue and doesnt get processed. If send few more messges let say 50
messages and it processes 40 messages and 10 msgs left on queue. I can send
more messages but not all get processed.
I am attaching config file. I have enabled producerFlowControl and removed
slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT.
On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <ch...@gmail.com>wrote:
> i said broker hangs bcs if i tried to send more messages or messages on
> queue doent get processed. i cannot even create queue or send message using
> webconsole.
>
> will try again and rhis time will moniter memory.
>
> On Feb 10, 2014 6:22 PM, "artnaseef" <ar...@artnaseef.com> wrote:<
>>
>> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
>> symptoms.
>>
>> The messages in the DLQ are failing at the consumer - either transactions
>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
>> client calling Session.recover().
>>
>> See here for more details:
>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>>
>> You are right about the webconsole -- it will most likely stop functioning
>> once the broker runs out of memory.
>>
>> >
>> >
>> > I have producerFlowControl to false. I seperated my producer and
>> > consumers.
>> > But still I was able to reproduce issue. For some time bunch of messages
>> > started pile up in queue. Then they started to endup in DLQ. In DLQ I
>> saw
>> > most of them have these property:
>> >
>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
>> > useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay
>> =
>> > 1000}, cause:null
>> >
>> > Does this thing has anything to do with the issue?
>> >
>> > I was not monitoring heap size when it got stuck. that will be next
>> thing
>> > to do. But I can hit webconsole from that activemq instance. So if it
>> had
>> > memory issues how come console is still working? isn't it same process?
>> >
>> > thanks,
>> > chirag
>> >
>> >
>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <ar...@artnaseef.com> wrote:
>> >
>> >> How about a stack trace on the client? Can you look for the consumer
>> >> threads
>> >> and see what they are doing?
>> >>
>> >> Note that with producers and consumers on the same connection, it's
>> >> possible
>> >> to reach a deadlock if producer-flow-control kicks in because the
>> entire
>> >> connection is blocked, not just the one producer. There are two ways
>> >> this
>> >> can happen. First, if the client produces and consumes the same
>> >> destination. Second, if the client is producing one destination and
>> >> consuming a second destination and another client is consuming the
>> first
>> >> and
>> >> producing to the second. Actually, more complicated possibilities
>> exist
>> >> as
>> >> well.
>> >>
>> >> Your best bet is to never consume and produce on the same connection so
>> >> that
>> >> consumption never blocks due to producer-flow-control. In that
>> >> scenario,
>> >> deadlocks won't happen because consumers can always consume.
>> >>
>> >> Back to stack traces - if the consuming threads are blocked waiting to
>> >> obtain a lock while producing threads hold that lock, that could
>> >> indicate
>> >> the problem.
>> >>
>> >> Anyway, slow consumption is quite often the cause of broker hangs, so
>> >> looking for slow consumption and understanding speeding consumption (or
>> >> slowing production) is important. If the broker JVM becomes completely
>> >> unresponsive, try connecting jconsole or visualvm before the broker
>> >> hangs
>> >> and then watch memory. If the JVM is running out of Heap or Permgen
>> >> space,
>> >> that would explain the hung broker.
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > If you reply to this email, your message will be added to the discussion
>> > below:
>> >
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
>> > To start a new topic under ActiveMQ - User, email
>> > ml-node+s2283324n2341805h3@n4.nabble.com
>> > To unsubscribe from ActiveMQ - User, visit
>> >
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
i said broker hangs bcs if i tried to send more messages or messages on
queue doent get processed. i cannot even create queue or send message using
webconsole.
will try again and rhis time will moniter memory.
On Feb 10, 2014 6:22 PM, "artnaseef" <ar...@artnaseef.com> wrote:<
>
> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
> symptoms.
>
> The messages in the DLQ are failing at the consumer - either transactions
> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
> client calling Session.recover().
>
> See here for more details:
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>
> You are right about the webconsole -- it will most likely stop functioning
> once the broker runs out of memory.
>
> >
> >
> > I have producerFlowControl to false. I seperated my producer and
> > consumers.
> > But still I was able to reproduce issue. For some time bunch of messages
> > started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
> > most of them have these property:
> >
> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
> > useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
> > 1000}, cause:null
> >
> > Does this thing has anything to do with the issue?
> >
> > I was not monitoring heap size when it got stuck. that will be next thing
> > to do. But I can hit webconsole from that activemq instance. So if it had
> > memory issues how come console is still working? isn't it same process?
> >
> > thanks,
> > chirag
> >
> >
> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <ar...@artnaseef.com> wrote:
> >
> >> How about a stack trace on the client? Can you look for the consumer
> >> threads
> >> and see what they are doing?
> >>
> >> Note that with producers and consumers on the same connection, it's
> >> possible
> >> to reach a deadlock if producer-flow-control kicks in because the entire
> >> connection is blocked, not just the one producer. There are two ways
> >> this
> >> can happen. First, if the client produces and consumes the same
> >> destination. Second, if the client is producing one destination and
> >> consuming a second destination and another client is consuming the first
> >> and
> >> producing to the second. Actually, more complicated possibilities exist
> >> as
> >> well.
> >>
> >> Your best bet is to never consume and produce on the same connection so
> >> that
> >> consumption never blocks due to producer-flow-control. In that
> >> scenario,
> >> deadlocks won't happen because consumers can always consume.
> >>
> >> Back to stack traces - if the consuming threads are blocked waiting to
> >> obtain a lock while producing threads hold that lock, that could
> >> indicate
> >> the problem.
> >>
> >> Anyway, slow consumption is quite often the cause of broker hangs, so
> >> looking for slow consumption and understanding speeding consumption (or
> >> slowing production) is important. If the broker JVM becomes completely
> >> unresponsive, try connecting jconsole or visualvm before the broker
> >> hangs
> >> and then watch memory. If the JVM is running out of Heap or Permgen
> >> space,
> >> that would explain the hung broker.
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> >
> > _______________________________________________
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
> > To start a new topic under ActiveMQ - User, email
> > ml-node+s2283324n2341805h3@n4.nabble.com
> > To unsubscribe from ActiveMQ - User, visit
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
Perhaps I misunderstood "Broker Hangs". It helps to have more detailed
symptoms.
The messages in the DLQ are failing at the consumer - either transactions
rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
client calling Session.recover().
See here for more details:
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
You are right about the webconsole -- it will most likely stop functioning
once the broker runs out of memory.
>
>
> I have producerFlowControl to false. I seperated my producer and
> consumers.
> But still I was able to reproduce issue. For some time bunch of messages
> started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
> most of them have these property:
>
> dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
> limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
> 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
> initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
> useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
> 1000}, cause:null
>
> Does this thing has anything to do with the issue?
>
> I was not monitoring heap size when it got stuck. that will be next thing
> to do. But I can hit webconsole from that activemq instance. So if it had
> memory issues how come console is still working? isn't it same process?
>
> thanks,
> chirag
>
>
> On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <ar...@artnaseef.com> wrote:
>
>> How about a stack trace on the client? Can you look for the consumer
>> threads
>> and see what they are doing?
>>
>> Note that with producers and consumers on the same connection, it's
>> possible
>> to reach a deadlock if producer-flow-control kicks in because the entire
>> connection is blocked, not just the one producer. There are two ways
>> this
>> can happen. First, if the client produces and consumes the same
>> destination. Second, if the client is producing one destination and
>> consuming a second destination and another client is consuming the first
>> and
>> producing to the second. Actually, more complicated possibilities exist
>> as
>> well.
>>
>> Your best bet is to never consume and produce on the same connection so
>> that
>> consumption never blocks due to producer-flow-control. In that
>> scenario,
>> deadlocks won't happen because consumers can always consume.
>>
>> Back to stack traces - if the consuming threads are blocked waiting to
>> obtain a lock while producing threads hold that lock, that could
>> indicate
>> the problem.
>>
>> Anyway, slow consumption is quite often the cause of broker hangs, so
>> looking for slow consumption and understanding speeding consumption (or
>> slowing production) is important. If the broker JVM becomes completely
>> unresponsive, try connecting jconsole or visualvm before the broker
>> hangs
>> and then watch memory. If the JVM is running out of Heap or Permgen
>> space,
>> that would explain the hung broker.
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805h3@n4.nabble.com
> To unsubscribe from ActiveMQ - User, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
I have producerFlowControl to false. I seperated my producer and consumers.
But still I was able to reproduce issue. For some time bunch of messages
started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
most of them have these property:
dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
1000}, cause:null
Does this thing has anything to do with the issue?
I was not monitoring heap size when it got stuck. that will be next thing
to do. But I can hit webconsole from that activemq instance. So if it had
memory issues how come console is still working? isn't it same process?
thanks,
chirag
On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <ar...@artnaseef.com> wrote:
> How about a stack trace on the client? Can you look for the consumer
> threads
> and see what they are doing?
>
> Note that with producers and consumers on the same connection, it's
> possible
> to reach a deadlock if producer-flow-control kicks in because the entire
> connection is blocked, not just the one producer. There are two ways this
> can happen. First, if the client produces and consumes the same
> destination. Second, if the client is producing one destination and
> consuming a second destination and another client is consuming the first
> and
> producing to the second. Actually, more complicated possibilities exist as
> well.
>
> Your best bet is to never consume and produce on the same connection so
> that
> consumption never blocks due to producer-flow-control. In that scenario,
> deadlocks won't happen because consumers can always consume.
>
> Back to stack traces - if the consuming threads are blocked waiting to
> obtain a lock while producing threads hold that lock, that could indicate
> the problem.
>
> Anyway, slow consumption is quite often the cause of broker hangs, so
> looking for slow consumption and understanding speeding consumption (or
> slowing production) is important. If the broker JVM becomes completely
> unresponsive, try connecting jconsole or visualvm before the broker hangs
> and then watch memory. If the JVM is running out of Heap or Permgen space,
> that would explain the hung broker.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
How about a stack trace on the client? Can you look for the consumer threads
and see what they are doing?
Note that with producers and consumers on the same connection, it's possible
to reach a deadlock if producer-flow-control kicks in because the entire
connection is blocked, not just the one producer. There are two ways this
can happen. First, if the client produces and consumes the same
destination. Second, if the client is producing one destination and
consuming a second destination and another client is consuming the first and
producing to the second. Actually, more complicated possibilities exist as
well.
Your best bet is to never consume and produce on the same connection so that
consumption never blocks due to producer-flow-control. In that scenario,
deadlocks won't happen because consumers can always consume.
Back to stack traces - if the consuming threads are blocked waiting to
obtain a lock while producing threads hold that lock, that could indicate
the problem.
Anyway, slow consumption is quite often the cause of broker hangs, so
looking for slow consumption and understanding speeding consumption (or
slowing production) is important. If the broker JVM becomes completely
unresponsive, try connecting jconsole or visualvm before the broker hangs
and then watch memory. If the JVM is running out of Heap or Permgen space,
that would explain the hung broker.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
I didnt had slowConsumerStrategybefore but I thought if connection are not
being closed properly this would help to shut those connection that's why I
added it.
Yes producers and consumer uses same connections.
I dont see any problem with heap size or queue usage size.
On Fri, Feb 7, 2014 at 3:05 PM, artnaseef <ar...@artnaseef.com> wrote:
> slowConsumerStrategy definitely sounds like it could lead to what you see
> since it force closes the connection.
>
> How about the memory and store usage of the queue? Do you see messages on
> the broker that producer-flow-control is kicking in?
>
> I need to read up on slowConsumerStrategy to see how it works.
>
> Are the same connections being used to both produce and consume?
>
> >
> >
> > I am not using cpp client.
> >
> > I checked producer. It sends message to broker and doesnt return. So it
> is
> > hanging at some point while sending. I see a message gets to borker but
> it
> > never gets picked by consumer.
> >
> > One more thing I have slowConsumerStrategy defined. Do you think it might
> > have anything to do with it.
> >
> > Here is example what I have:
> >
> > <slowConsumerStrategy>
> > <abortSlowConsumerStrategy
> > abortConnection="true"/>
> > </slowConsumerStrategy>
> >
> > More confusing thing is since yesterday consumer count is stuck on 100.
> It
> > is not going down and it's not consuming any messages from queue.
> >
> >
> > On Thu, Feb 6, 2014 at 7:48 PM, artnaseef <ar...@artnaseef.com> wrote:
> >
> >> Hmm, that is very interesting. Is the client a C++ (activemq-cpp)
> >> client?
> >>
> >> So, first off, the "Connection reset" error means a client is losing
> >> connection to the broker without cleaning up first - not sure why, but
> >> that itself isn't indicative of your problem. Keep it in mind because
> >> it
> >> may be triggering the problem, but it's not the cause.
> >>
> >> This looks like it could be a bug in the producer, but that's premature
> >> to
> >> assert. Can you look at the producer code to see how it handles failed
> >> sends? It looks like the producer may be getting stuck on a failure.
> >> Check how the producer handles connection to the broker when the
> >> connection fails. If the client is using the failover transport, that
> >> shouldn't be an issue - although it depends on the transport settings.
> >>
> >>
> >> >
> >> >
> >> > Thanks for reply.
> >> >
> >> > -I checked consumer and produce count on queue and they are 99(c) and
> >> 7(p)
> >> > respectively.
> >> > - I see this warning messgage randomly in broker logs: "2014-02-06
> >> > 13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669
> >> failed:
> >> > java.net.SocketException: Connection reset |
> >> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> >> > Transport: tcp:///[ip]:63669@61616"
> >> >
> >> > and on client side I see following debug messages:
> >> >
> >> > 14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
> >> > message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
> >> > messageId = null, originalDestination = null, originalTransactionId =
> >> > null,
> >> > producerId = null, destination = null, transactionId = null,
> >> expiration =
> >> > 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
> >> > correlationId = null, replyTo = null, persistent = false, type = null,
> >> > priority = 0, groupID = null, groupSequence = 0, targetConsumerId =
> >> null,
> >> > compressed = false, userID = null, content = null,
> >> marshalledProperties =
> >> > null, dataStructure = null, redeliveryCounter = 0, size = 0,
> >> properties =
> >> > null, readOnlyProperties = false, readOnlyBody = false, droppable =
> >> false,
> >> > text = <?xml version="1.0" encoding="ASCII" ?><Audit...uditMessage>}
> >> > 14:15:33.803
> >> >
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> >> > DEBUG org.apache.activemq.ActiveMQSession -
> >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >> > 14:15:34.803
> >> >
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> >> > DEBUG org.apache.activemq.ActiveMQSession -
> >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >> > 14:15:35.803
> >> >
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> >> > DEBUG org.apache.activemq.ActiveMQSession -
> >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >> > 14:15:36.803
> >> >
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> >> > DEBUG org.apache.activemq.ActiveMQSession -
> >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >> > 14:15:37.803
> >> >
> >>
> [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> >> > DEBUG org.apache.activemq.ActiveMQSession -
> >> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >> >
> >> > and it just keep on going and never stops. I am not worried about
> >> > healthcheck stuff right now. It's just there and printing logs so
> >> ignore
> >> > that part.
> >> >
> >> > this behavior happens when activemq borker is frozen. I mean queue
> >> count
> >> > is
> >> > stuck on 230.
> >> >
> >> > thanks,
> >> > chirag
> >> >
> >> >
> >> > On Wed, Feb 5, 2014 at 8:07 PM, artnaseef <ar...@artnaseef.com> wrote:
> >> >
> >> >> There are many things to check.
> >> >>
> >> >> First off, I have a little confusion. You metion no log messages on
> >> the
> >> >> client or server and then show log messages. By server, do you mean
> >> the
> >> >> broker or something else?
> >> >>
> >> >> Troubleshooting steps:
> >> >> * When flow stops, check the broker for active client connections via
> >> >> the
> >> >> webconsole or JMX.
> >> >> * Check consumer and producer counts for the destinations.
> >> >> * If the failover transport is being used, be sure to use a
> >> >> ConnectionListener and TransportListener to receive notifications of
> >> >> lost
> >> >> connection to the broker. Note that you should get a log message if
> >> the
> >> >> failover transport loses connection, so this may not be necessary.
> >> >> * Check broker JVM memory (especially Heap and PermGen)
> >> >>
> >> >> Also, if the broker is running out of memory due to having too many
> >> >> messages
> >> >> (i.e. slow consumer problem), that could cause the broker to stop
> >> >> functioning.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > If you reply to this email, your message will be added to the
> >> discussion
> >> > below:
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html
> >> > To start a new topic under ActiveMQ - User, email
> >> > ml-node+s2283324n2341805h3@n4.nabble.com
> >> > To unsubscribe from ActiveMQ - User, visit
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> >
> > _______________________________________________
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677642.html
> > To start a new topic under ActiveMQ - User, email
> > ml-node+s2283324n2341805h3@n4.nabble.com
> > To unsubscribe from ActiveMQ - User, visit
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677643.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
slowConsumerStrategy definitely sounds like it could lead to what you see
since it force closes the connection.
How about the memory and store usage of the queue? Do you see messages on
the broker that producer-flow-control is kicking in?
I need to read up on slowConsumerStrategy to see how it works.
Are the same connections being used to both produce and consume?
>
>
> I am not using cpp client.
>
> I checked producer. It sends message to broker and doesnt return. So it is
> hanging at some point while sending. I see a message gets to borker but it
> never gets picked by consumer.
>
> One more thing I have slowConsumerStrategy defined. Do you think it might
> have anything to do with it.
>
> Here is example what I have:
>
> <slowConsumerStrategy>
> <abortSlowConsumerStrategy
> abortConnection="true"/>
> </slowConsumerStrategy>
>
> More confusing thing is since yesterday consumer count is stuck on 100. It
> is not going down and it's not consuming any messages from queue.
>
>
> On Thu, Feb 6, 2014 at 7:48 PM, artnaseef <ar...@artnaseef.com> wrote:
>
>> Hmm, that is very interesting. Is the client a C++ (activemq-cpp)
>> client?
>>
>> So, first off, the "Connection reset" error means a client is losing
>> connection to the broker without cleaning up first - not sure why, but
>> that itself isn't indicative of your problem. Keep it in mind because
>> it
>> may be triggering the problem, but it's not the cause.
>>
>> This looks like it could be a bug in the producer, but that's premature
>> to
>> assert. Can you look at the producer code to see how it handles failed
>> sends? It looks like the producer may be getting stuck on a failure.
>> Check how the producer handles connection to the broker when the
>> connection fails. If the client is using the failover transport, that
>> shouldn't be an issue - although it depends on the transport settings.
>>
>>
>> >
>> >
>> > Thanks for reply.
>> >
>> > -I checked consumer and produce count on queue and they are 99(c) and
>> 7(p)
>> > respectively.
>> > - I see this warning messgage randomly in broker logs: "2014-02-06
>> > 13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669
>> failed:
>> > java.net.SocketException: Connection reset |
>> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
>> > Transport: tcp:///[ip]:63669@61616"
>> >
>> > and on client side I see following debug messages:
>> >
>> > 14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
>> > message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
>> > messageId = null, originalDestination = null, originalTransactionId =
>> > null,
>> > producerId = null, destination = null, transactionId = null,
>> expiration =
>> > 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
>> > correlationId = null, replyTo = null, persistent = false, type = null,
>> > priority = 0, groupID = null, groupSequence = 0, targetConsumerId =
>> null,
>> > compressed = false, userID = null, content = null,
>> marshalledProperties =
>> > null, dataStructure = null, redeliveryCounter = 0, size = 0,
>> properties =
>> > null, readOnlyProperties = false, readOnlyBody = false, droppable =
>> false,
>> > text = <?xml version="1.0" encoding="ASCII" ?><Audit...uditMessage>}
>> > 14:15:33.803
>> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
>> > DEBUG org.apache.activemq.ActiveMQSession -
>> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>> > 14:15:34.803
>> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
>> > DEBUG org.apache.activemq.ActiveMQSession -
>> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>> > 14:15:35.803
>> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
>> > DEBUG org.apache.activemq.ActiveMQSession -
>> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>> > 14:15:36.803
>> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
>> > DEBUG org.apache.activemq.ActiveMQSession -
>> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>> > 14:15:37.803
>> >
>> [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
>> > DEBUG org.apache.activemq.ActiveMQSession -
>> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>> >
>> > and it just keep on going and never stops. I am not worried about
>> > healthcheck stuff right now. It's just there and printing logs so
>> ignore
>> > that part.
>> >
>> > this behavior happens when activemq borker is frozen. I mean queue
>> count
>> > is
>> > stuck on 230.
>> >
>> > thanks,
>> > chirag
>> >
>> >
>> > On Wed, Feb 5, 2014 at 8:07 PM, artnaseef <ar...@artnaseef.com> wrote:
>> >
>> >> There are many things to check.
>> >>
>> >> First off, I have a little confusion. You metion no log messages on
>> the
>> >> client or server and then show log messages. By server, do you mean
>> the
>> >> broker or something else?
>> >>
>> >> Troubleshooting steps:
>> >> * When flow stops, check the broker for active client connections via
>> >> the
>> >> webconsole or JMX.
>> >> * Check consumer and producer counts for the destinations.
>> >> * If the failover transport is being used, be sure to use a
>> >> ConnectionListener and TransportListener to receive notifications of
>> >> lost
>> >> connection to the broker. Note that you should get a log message if
>> the
>> >> failover transport loses connection, so this may not be necessary.
>> >> * Check broker JVM memory (especially Heap and PermGen)
>> >>
>> >> Also, if the broker is running out of memory due to having too many
>> >> messages
>> >> (i.e. slow consumer problem), that could cause the broker to stop
>> >> functioning.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > If you reply to this email, your message will be added to the
>> discussion
>> > below:
>> >
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html
>> > To start a new topic under ActiveMQ - User, email
>> > ml-node+s2283324n2341805h3@n4.nabble.com
>> > To unsubscribe from ActiveMQ - User, visit
>> >
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677642.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805h3@n4.nabble.com
> To unsubscribe from ActiveMQ - User, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677643.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
I am not using cpp client.
I checked producer. It sends message to broker and doesnt return. So it is
hanging at some point while sending. I see a message gets to borker but it
never gets picked by consumer.
One more thing I have slowConsumerStrategy defined. Do you think it might
have anything to do with it.
Here is example what I have:
<slowConsumerStrategy>
<abortSlowConsumerStrategy abortConnection="true"/>
</slowConsumerStrategy>
More confusing thing is since yesterday consumer count is stuck on 100. It
is not going down and it's not consuming any messages from queue.
On Thu, Feb 6, 2014 at 7:48 PM, artnaseef <ar...@artnaseef.com> wrote:
> Hmm, that is very interesting. Is the client a C++ (activemq-cpp) client?
>
> So, first off, the "Connection reset" error means a client is losing
> connection to the broker without cleaning up first - not sure why, but
> that itself isn't indicative of your problem. Keep it in mind because it
> may be triggering the problem, but it's not the cause.
>
> This looks like it could be a bug in the producer, but that's premature to
> assert. Can you look at the producer code to see how it handles failed
> sends? It looks like the producer may be getting stuck on a failure.
> Check how the producer handles connection to the broker when the
> connection fails. If the client is using the failover transport, that
> shouldn't be an issue - although it depends on the transport settings.
>
>
> >
> >
> > Thanks for reply.
> >
> > -I checked consumer and produce count on queue and they are 99(c) and
> 7(p)
> > respectively.
> > - I see this warning messgage randomly in broker logs: "2014-02-06
> > 13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669 failed:
> > java.net.SocketException: Connection reset |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///[ip]:63669@61616"
> >
> > and on client side I see following debug messages:
> >
> > 14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
> > message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
> > messageId = null, originalDestination = null, originalTransactionId =
> > null,
> > producerId = null, destination = null, transactionId = null, expiration =
> > 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
> > correlationId = null, replyTo = null, persistent = false, type = null,
> > priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null,
> > compressed = false, userID = null, content = null, marshalledProperties =
> > null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> > null, readOnlyProperties = false, readOnlyBody = false, droppable =
> false,
> > text = <?xml version="1.0" encoding="ASCII" ?><Audit...uditMessage>}
> > 14:15:33.803
> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> > DEBUG org.apache.activemq.ActiveMQSession -
> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> > 14:15:34.803
> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> > DEBUG org.apache.activemq.ActiveMQSession -
> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> > 14:15:35.803
> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> > DEBUG org.apache.activemq.ActiveMQSession -
> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> > 14:15:36.803
> > [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> > DEBUG org.apache.activemq.ActiveMQSession -
> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> > 14:15:37.803
> >
> [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> > DEBUG org.apache.activemq.ActiveMQSession -
> > ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> >
> > and it just keep on going and never stops. I am not worried about
> > healthcheck stuff right now. It's just there and printing logs so ignore
> > that part.
> >
> > this behavior happens when activemq borker is frozen. I mean queue count
> > is
> > stuck on 230.
> >
> > thanks,
> > chirag
> >
> >
> > On Wed, Feb 5, 2014 at 8:07 PM, artnaseef <ar...@artnaseef.com> wrote:
> >
> >> There are many things to check.
> >>
> >> First off, I have a little confusion. You metion no log messages on the
> >> client or server and then show log messages. By server, do you mean the
> >> broker or something else?
> >>
> >> Troubleshooting steps:
> >> * When flow stops, check the broker for active client connections via
> >> the
> >> webconsole or JMX.
> >> * Check consumer and producer counts for the destinations.
> >> * If the failover transport is being used, be sure to use a
> >> ConnectionListener and TransportListener to receive notifications of
> >> lost
> >> connection to the broker. Note that you should get a log message if the
> >> failover transport loses connection, so this may not be necessary.
> >> * Check broker JVM memory (especially Heap and PermGen)
> >>
> >> Also, if the broker is running out of memory due to having too many
> >> messages
> >> (i.e. slow consumer problem), that could cause the broker to stop
> >> functioning.
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> >
> > _______________________________________________
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html
> > To start a new topic under ActiveMQ - User, email
> > ml-node+s2283324n2341805h3@n4.nabble.com
> > To unsubscribe from ActiveMQ - User, visit
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
Hmm, that is very interesting. Is the client a C++ (activemq-cpp) client?
So, first off, the "Connection reset" error means a client is losing
connection to the broker without cleaning up first - not sure why, but
that itself isn't indicative of your problem. Keep it in mind because it
may be triggering the problem, but it's not the cause.
This looks like it could be a bug in the producer, but that's premature to
assert. Can you look at the producer code to see how it handles failed
sends? It looks like the producer may be getting stuck on a failure.
Check how the producer handles connection to the broker when the
connection fails. If the client is using the failover transport, that
shouldn't be an issue - although it depends on the transport settings.
>
>
> Thanks for reply.
>
> -I checked consumer and produce count on queue and they are 99(c) and 7(p)
> respectively.
> - I see this warning messgage randomly in broker logs: "2014-02-06
> 13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669 failed:
> java.net.SocketException: Connection reset |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///[ip]:63669@61616"
>
> and on client side I see following debug messages:
>
> 14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
> message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
> messageId = null, originalDestination = null, originalTransactionId =
> null,
> producerId = null, destination = null, transactionId = null, expiration =
> 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
> correlationId = null, replyTo = null, persistent = false, type = null,
> priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null,
> compressed = false, userID = null, content = null, marshalledProperties =
> null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> null, readOnlyProperties = false, readOnlyBody = false, droppable = false,
> text = <?xml version="1.0" encoding="ASCII" ?><Audit...uditMessage>}
> 14:15:33.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:34.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:35.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:36.803
> [com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
> 14:15:37.803
> [com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
> DEBUG org.apache.activemq.ActiveMQSession -
> ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
>
> and it just keep on going and never stops. I am not worried about
> healthcheck stuff right now. It's just there and printing logs so ignore
> that part.
>
> this behavior happens when activemq borker is frozen. I mean queue count
> is
> stuck on 230.
>
> thanks,
> chirag
>
>
> On Wed, Feb 5, 2014 at 8:07 PM, artnaseef <ar...@artnaseef.com> wrote:
>
>> There are many things to check.
>>
>> First off, I have a little confusion. You metion no log messages on the
>> client or server and then show log messages. By server, do you mean the
>> broker or something else?
>>
>> Troubleshooting steps:
>> * When flow stops, check the broker for active client connections via
>> the
>> webconsole or JMX.
>> * Check consumer and producer counts for the destinations.
>> * If the failover transport is being used, be sure to use a
>> ConnectionListener and TransportListener to receive notifications of
>> lost
>> connection to the broker. Note that you should get a log message if the
>> failover transport loses connection, so this may not be necessary.
>> * Check broker JVM memory (especially Heap and PermGen)
>>
>> Also, if the broker is running out of memory due to having too many
>> messages
>> (i.e. slow consumer problem), that could cause the broker to stop
>> functioning.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677565.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805h3@n4.nabble.com
> To unsubscribe from ActiveMQ - User, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677600.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Hangs after some time - or does nothing.
Posted by Chirag Pujara <ch...@gmail.com>.
Thanks for reply.
-I checked consumer and produce count on queue and they are 99(c) and 7(p)
respectively.
- I see this warning messgage randomly in broker logs: "2014-02-06
13:53:28,910 | WARN | Transport Connection to: tcp://[ip]:63669 failed:
java.net.SocketException: Connection reset |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///[ip]:63669@61616"
and on client side I see following debug messages:
14:15:32.819 [main] DEBUG o.s.jms.core.JmsTemplate - Sending created
message: ActiveMQTextMessage {commandId = 0, responseRequired = false,
messageId = null, originalDestination = null, originalTransactionId = null,
producerId = null, destination = null, transactionId = null, expiration =
0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0,
correlationId = null, replyTo = null, persistent = false, type = null,
priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null,
compressed = false, userID = null, content = null, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
null, readOnlyProperties = false, readOnlyBody = false, droppable = false,
text = <?xml version="1.0" encoding="ASCII" ?><Audit...uditMessage>}
14:15:33.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:34.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:35.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:36.803
[com.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
14:15:37.803
[com.gwx.commons.jms.healthcheck.listener.DefaultMessageListenerContainer-1]
DEBUG org.apache.activemq.ActiveMQSession -
ID:C-CP-64386-1391717730167-3:1:1 Transaction Commit :null
and it just keep on going and never stops. I am not worried about
healthcheck stuff right now. It's just there and printing logs so ignore
that part.
this behavior happens when activemq borker is frozen. I mean queue count is
stuck on 230.
thanks,
chirag
On Wed, Feb 5, 2014 at 8:07 PM, artnaseef <ar...@artnaseef.com> wrote:
> There are many things to check.
>
> First off, I have a little confusion. You metion no log messages on the
> client or server and then show log messages. By server, do you mean the
> broker or something else?
>
> Troubleshooting steps:
> * When flow stops, check the broker for active client connections via the
> webconsole or JMX.
> * Check consumer and producer counts for the destinations.
> * If the failover transport is being used, be sure to use a
> ConnectionListener and TransportListener to receive notifications of lost
> connection to the broker. Note that you should get a log message if the
> failover transport loses connection, so this may not be necessary.
> * Check broker JVM memory (especially Heap and PermGen)
>
> Also, if the broker is running out of memory due to having too many
> messages
> (i.e. slow consumer problem), that could cause the broker to stop
> functioning.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
There are many things to check.
First off, I have a little confusion. You metion no log messages on the
client or server and then show log messages. By server, do you mean the
broker or something else?
Troubleshooting steps:
* When flow stops, check the broker for active client connections via the
webconsole or JMX.
* Check consumer and producer counts for the destinations.
* If the failover transport is being used, be sure to use a
ConnectionListener and TransportListener to receive notifications of lost
connection to the broker. Note that you should get a log message if the
failover transport loses connection, so this may not be necessary.
* Check broker JVM memory (especially Heap and PermGen)
Also, if the broker is running out of memory due to having too many messages
(i.e. slow consumer problem), that could cause the broker to stop
functioning.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677512.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Broker Hangs after some time - or does nothing.
Posted by artnaseef <ar...@artnaseef.com>.
Ah, it's possible to change the subject. Got it.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677514.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ to Apollo
Posted by artnaseef <ar...@artnaseef.com>.
OK, something is very wrong here. On Nabble, I clearly click on "Broker
Hangs after some time - or does nothing." but then, reply shows Subject "Re:
ActiveMQ to Apollo".
Very odd.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ to Apollo
Posted by artnaseef <ar...@artnaseef.com>.
Hey Erik - I think you inadvertently hijacked another thread. Please repost
your question by creating a new thread. I would do it for you, but then it
won't be linked to you.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677511.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
ActiveMQ to Apollo
Posted by "Erik S. Peterson" <pe...@ldschurch.org>.
Hi, I'm new to this forum and have a few questions as we are evaluating ActiveMQ, Apollo, and other message brokers. Please redirect me if there is a better place for Apollo questions.
1) Apollo is the new ActiveMQ yes? If we want to upgrade from ActiveMQ 1.5...the only upgrade path is Apollo right?
2) Apollo seems immature at this stage. But is it being used in highly critical production environments? If not, when do you anticipate it being ready for such?
3) Generally, where is Apollo strong and where is it weak. A vague question, but in comparison to ActiveMQ, where does Apollo really excel or fall short?
4) What companies are providing commercial support for Apollo or backing its development?
5) I was pleased when seeing that Apollo is leveraging Scala...only to find out that some major architecture changes had taken place and that Scala doesn't appear to actually be well leveraged. For example, rather than using Akka, Apollo uses HawtDispatch...Akka used HawtDispatch at one time but dropped this library 3 years ago in favor of a native, faster, generally more capable design. It seams that a better direction would be for Apollo to wrap Akka and add other ActiveMQ items. But please share your design thinking and future direction.
Thank you!
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.