ActiveMq consumer intermittently hanging after reconnect

[wayn23 <wa...@gwl.ca>]

I have run into an intermittent problem with ActiveMQ V5.9.0. The
intermittent 
was tracked down to being a connection error that does not get properly 
recovered. 

I have recreated the problem in a very small example by creating a Camel 
consumer route that retrieves messages from a JMS queue and writes the text 
message to a log.  

	from(TEST_QUEUE).routeId(TEST_QUEUE).convertBodyTo(String.class)
				.log(LoggingLevel.INFO, loggerName, "Request Received ");

  The connection string used is 
 
failover:(tcp://serverd05.company.com:26093?keepAlive=true)?jms.prefetchPolicy.all=0 
  
Please note that polling is being used when the prefetch size is set to 0.

I then use Hawtio on the ActiveMQ broker to add some text messages to the
queue. It works fine. 

I then stop the Client Connector on the ActiveMQ broker side to simulate a 
broken connection on the consumer side. (The broker must be remote to the 
consumer or the error will not occur.)  The client logs show that an 
EOFException is caught and the connection is reestablished. The connection
also 
appears on the ActiveMQ broker. it only looks good though. Sending new text 
messages to the queue will not be processed. They will just sit there. 
There 
are no errors or warnings logged on either the consumer or broker servers. 
Restarting the consumer will cause the messages to get processed. The 
reconnection only intermittently fails. I find is much more like to occur if
10 
minutes pass from the previous message being processed.

I turned on trace=yes in the connection string and found the root cause is
the 
PullMessage commands are occasionally not being issued after a reconnect. 
On 
cases where it works, the log shows that the PullMessage commands do
reinstate. 
I have included a log file of the failed reconnect, where the PullMessage 
commands are not restarted.  Simply restarting the consumer will resolve the 
problem.
FailedReconnect.log
<http://activemq.2283324.n4.nabble.com/file/n4684226/FailedReconnect.log>  

I am wondering if this might be a race condition, as this problem only
showed up 
when the consumer was on fast servers




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-consumer-intermittently-hanging-after-reconnect-tp4684226.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMq consumer intermittently hanging after reconnect

[wayn23 <wa...@gwl.ca>]

I have opened JIRA AMQ-5313

Thanks




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-consumer-intermittently-hanging-after-reconnect-tp4684226p4684258.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMq consumer intermittently hanging after reconnect

[Gary Tully <ga...@gmail.com>]

Please raise a jira issue to track this. we may need to track pull requests
or it could be some sort of race. In any event it is a bug.
On 7 Aug 2014 01:09, "wayn23" <wa...@gwl.ca> wrote:

> I have run into an intermittent problem with ActiveMQ V5.9.0. The
> intermittent
> was tracked down to being a connection error that does not get properly
> recovered.
>
> I have recreated the problem in a very small example by creating a Camel
> consumer route that retrieves messages from a JMS queue and writes the text
> message to a log.
>
>         from(TEST_QUEUE).routeId(TEST_QUEUE).convertBodyTo(String.class)
>                                 .log(LoggingLevel.INFO, loggerName,
> "Request Received ");
>
>   The connection string used is
>
> failover:(tcp://
> serverd05.company.com:26093?keepAlive=true)?jms.prefetchPolicy.all=0
>
> Please note that polling is being used when the prefetch size is set to 0.
>
> I then use Hawtio on the ActiveMQ broker to add some text messages to the
> queue. It works fine.
>
> I then stop the Client Connector on the ActiveMQ broker side to simulate a
> broken connection on the consumer side. (The broker must be remote to the
> consumer or the error will not occur.)  The client logs show that an
> EOFException is caught and the connection is reestablished. The connection
> also
> appears on the ActiveMQ broker. it only looks good though. Sending new text
> messages to the queue will not be processed. They will just sit there.
> There
> are no errors or warnings logged on either the consumer or broker servers.
> Restarting the consumer will cause the messages to get processed. The
> reconnection only intermittently fails. I find is much more like to occur
> if
> 10
> minutes pass from the previous message being processed.
>
> I turned on trace=yes in the connection string and found the root cause is
> the
> PullMessage commands are occasionally not being issued after a reconnect.
> On
> cases where it works, the log shows that the PullMessage commands do
> reinstate.
> I have included a log file of the failed reconnect, where the PullMessage
> commands are not restarted.  Simply restarting the consumer will resolve
> the
> problem.
> FailedReconnect.log
> <http://activemq.2283324.n4.nabble.com/file/n4684226/FailedReconnect.log>
>
> I am wondering if this might be a race condition, as this problem only
> showed up
> when the consumer was on fast servers
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMq-consumer-intermittently-hanging-after-reconnect-tp4684226.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: ActiveMq consumer intermittently hanging after reconnect

[artnaseef <ar...@artnaseef.com>]

That does sound like a race condition based on the description provided.  It
also sounds like you have almost solved it - if you can finish tracking it
down and provide a patch, that would be most welcome.

Good work!



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-consumer-intermittently-hanging-after-reconnect-tp4684226p4684268.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.