[jira] [Created] (AMQ-7337) low consumer with prefetch 0 fails to re-establish message pull state after failover

["Jonathan Swenson (Jira)" <ji...@apache.org>]

Jonathan Swenson created AMQ-7337:
-------------------------------------

             Summary: low consumer with prefetch 0 fails to re-establish message pull state after failover
                 Key: AMQ-7337
                 URL: https://issues.apache.org/jira/browse/AMQ-7337
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.15.9, 5.15.2
         Environment: Reproduction program: Oracle JDK 8 on Mac OSX 

Original finding: Oracle JDK8 on Ubuntu
            Reporter: Jonathan Swenson


Periodically, I have a prefetch 0 consumer that stops receiving messages from the broker when it is idle for an extended period of time before a broker failover occurs.
 Upon a re-connection, the connection (and consumer) is successfully re-established with the new broker, but the consumer will never receive messages again.
 There are many WARN lines in the logs. However, none of these correspond with the session for the consumer that is not receiving messages. These all correspond to subscriptions that have been closed:

 
{code:java}
2019-11-05 11:24:14,490 [ActiveMQ Transport: tcp:///127.0.0.1:58742@61616] WARN - Async error occurred: java.lang.IllegalArgumentException: The subscription does not exist: ID:local-58699-1572981850004-8:1:1799:1{code}

 Other consumers on this same connection (also configured with prefetch 0) will successfully receive messages.

System config:
 Master / Slave broker configuration
 Shared connection between multiple consumers on multiple queues.
 Connection configured with `jms.prefetchPolicy.all=0`
 Transacted Consumers receive messages with a timeout of zero (blocking until message is sent)
 Consumers are short lived (only receive a single message before closing)
 Discovered on activemq client 5.15.2, reproduced on 5.15.9


 I created a few test cases which I believe highlight the problem I'm seeing (attached)


 For simplicity, the failover is modeled as stopping and starting up a new BrokerService.

Notably, there appears to be a message cache of variable size on the  ConnectionStateTracker that contains MessagePull commands that will be sent to the new broker when a failover happens. 
 If a consumer waits for a message for an extended period of time, its MessagePull command will get ejected from this cache if there are enough other MessagePulls from other consumers that fill the cache.
 It looks like a MessagePull command is sent every time a receive is called on a Consumer that is configured with prefetch 0, but they are de-duplicated (by consumer id) by the ConnectionStateTracker.
 If there are 328 (with the 128k default cache size) consumers created which each issue at least a single receive, the entire cache will have cycled and anything in the cache before will have gotten evicted leaving any other consumers waiting for messages indefinitely.
 In these test cases, we see a large number of consumers and a saturated messageCache with MessagePull commands that are no longer relevant because the consumer has since closed. Upon re-connection to a new broker, these MessagePull commands are flushed, and the Broker rejects them with the above error because the session has since been closed. Perhaps these commands can be removed from the cache when the consumer closes. However, this same problem could occur with the same number of "active" consumers.


 Seeking to work around this, I tried using a timed receive model (waiting for a second before giving up and trying again), however even the timed receive seems to hang if a failover occurs after the consumer's MessagePull is ejected from the cache as a prefetch 0 consumer seems to rely on the broker to notify it when the timeout happens, but if a failover occurs, the new broker won't be able to notify it of the expiration. However, using a shorter receive timeout reduces the windows of time where this can occur (as enough MessagePulls must be sent to overflow the cache in a small time window).


 Moving a to re-used consumer seems to avoid this problem in this simple test case (receiving all messages on the same consumer) as there would only be a single additional consumer and not overflow this cache.


 Bumping up the cache size seems to defer the problem, allowing for more consumers to be used before a long running consumer might hang.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)