[jira] [Updated] (AMQ-4556) Store based cursor "forgets" messages

["Matt Pavlovich (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Pavlovich updated AMQ-4556:
--------------------------------
    Labels: close-pending  (was: )

> Store based cursor "forgets" messages
> -------------------------------------
>
>                 Key: AMQ-4556
>                 URL: https://issues.apache.org/jira/browse/AMQ-4556
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.7.0, 5.8.0
>         Environment: Durable queues with JDBC persistent store in a master/slave configuration with NIO transport
>            Reporter: SunGard Global Services Germany
>            Priority: Major
>              Labels: close-pending
>             Fix For: NEEDS_REVIEW
>
>         Attachments: ActiveMQClient.java, ActiveMQClient2.java, activemq.xml
>
>
> This issue seems to relate to AMQ-2009 and the referenced articles there.
> h3. The issue:
> After a give period of time, the broker seems to "forget" about received messages on a queue.
> Receiving of new messages and delivering them is unaffected. Just some are lost in the store.
> Checking the DB tables, the missing messages do exists there.
> ActiveMQ statistics counters are also aware of them and count those messages as "pending" in the ActiveMQ console (first colummn). But open the details of a queue in ActiveMQ console do not show them at all.
> h3. Analysis so far
> We tried several settings for the queues which had no impact on the issue:
> * Activating/Deactivating useCache
> * Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax the issue a little bit)
> * KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
> Also some characteristics about affected/not affected queues might help to analyse this further:
> * We have one queue with just one producer and many consumers whereas the message is quite small (just headers): No problem on that queue even after thousend of messages
> * Queues with multiple producers and consumers and payloaded text-messages seem to be unaffected - Maybe the JDBC persistent store trottels the processing enough to "solve" the issue
> * Queues with multiple producers and consumers and small messages (just headers) seem to "enable" the issue. Even with few messages the issue appears
> h3. "Recovering" lost messages
> Shutdown the (master) broker and restart it. With failover transport this happens transparent for the clients.
> On master startup, all messages from the store (DB) are scanned and "rediscovered"
> h3. Workaround
> Use another cursor - VM seems to be fine. Additional settings might be required to handle shotcommings of the VM cursor (max memory, queue memory etc.)
> {code:xml}
>                 <policyEntry queue=">" ...>
>                   <pendingQueuePolicy>
>                     <vmQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
> {code}
> h3. Test code
> I was not able to create a self-contained unit test.
> But the attached code can be used to reproduce the issue quite reliable.
> It sends 3000 messages. WIth the following settings it will fail/will work correctly:
> || Message Size || ActiveMQ cursor || Result ||
> | 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages delivered|
> | 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages delivered|
> | remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
> | remove comment from line 20 | store based Cursor| (x) - Some (~0,5% messages lost)|
> For completness also the used activemq-broker.xml was attached - This is the default one with the JDBC persistent store added.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)