[jira] [Created] (AMQ-4556) Store based cursor "forgets" messages

["SunGard Global Services Germany (JIRA)" <ji...@apache.org>]

SunGard Global Services Germany created AMQ-4556:
----------------------------------------------------

             Summary: Store based cursor "forgets" messages
                 Key: AMQ-4556
                 URL: https://issues.apache.org/jira/browse/AMQ-4556
             Project: ActiveMQ
          Issue Type: Bug
          Components: Message Store
    Affects Versions: 5.8.0, 5.7.0
         Environment: Durable queues with JDBC persistent store in a master/slave configuration with NIO transport
            Reporter: SunGard Global Services Germany
            Priority: Critical


This issue seems to relate to AMQ-2009 and the referenced articles there.

h3. The issue:
After a give period of time, the broker seems to "forget" about received messages on a queue.
Receiving of new messages and delivering them is unaffected. Just some are lost in the store.
Checking the DB tables, the missing messages do exists there.
ActiveMQ statistics counters are also aware of them and count those messages as "pending" in the ActiveMQ console (first colummn). But open the details of a queue in ActiveMQ console do not show them at all.

h3. Analysis so far
We tried several settings for the queues which had no impact on the issue:
* Activating/Deactivating useCache
* Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax the issue a little bit)
* KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected

Also some characteristics about affected/not affected queues might help to analyse this further:
* We have one queue with just one producer and many consumers whereas the message is quite small (just headers): No problem on that queue even after thousend of messages
* Queues with multiple producers and consumers and payloaded text-messages seem to be unaffected - Maybe the JDBC persistent store trottels the processing enough to "solve" the issue
* Queues with multiple producers and consumers and small messages (just headers) seem to "enable" the issue. Even with few messages the issue appears

h3. "Recovering" lost messages
Shutdown the (master) broker and restart it. With failover transport this happens transparent for the clients.
On master startup, all messages from the store (DB) are scanned and "rediscovered"

h3. Workaround
Use another cursor - VM seems to be fine. Additional settings might be required to handle shotcommings of the VM cursor (max memory, queue memory etc.)
{code:xml}
                <policyEntry queue=">" ...>
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
{code}

h3. Test code
I was not able to create a self-contained unit test.
But the attached code can be used to reproduce the issue quite reliable.
It sends 3000 messages. WIth the following settings it will fail/will work correctly:
|| Message Size || ActiveMQ cursor || Result ||
| 1 MB (leave line 20 as it is) | vmQueueCursor | (/) |
| 1 MB (leave line 20 as it is) | store based Cursor| (/) |
| remove comment from line 20 | vmQueueCursor| (/) |
| remove comment from line 20 | store based Cursor| (x) |

For completness also the used activemq-broker.xml was attached - This is the default one with the JDBC persistent store added.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira