[jira] [Issue Comment Deleted] (AMQ-5249) "cursor got duplicate" error after upgrade

["Alex Chernyakhovsky (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Chernyakhovsky updated AMQ-5249:
-------------------------------------
    Comment: was deleted

(was: Gary,

I will check the unit test module but I am afraid it might prove hard to replicate our environment.
Maybe it will be more productive if I work with you or someone assigned to this issue to understand what exactly is happening and then proceed with the test.

Here is some info:
1) Our production system consists of about half a dozen of servers running clients producing messages, then another set of about 10 consuming the load while producing some messages for consumption by themselves and another set of about 5 servers running apache storm. (The numbers change as we scale). 8 queues are created across the board.
2) We've ran into some reliability issues with the network of brokers while using AMQ 5.8.0 (transient) so we resorted to a single beefed up server.
3) Now we are on 5.10.0 and it is able to keep up with the current load but the load tends to grow and we would want to have some failover capability too.
4) I am running a load test for the same configuration as in production with 5, 7, 6 servers and again a beefy AMQ 5.10.0 server works fine.
5) When I add another AMQ server using static connector everything seems to work fine under lighter load. As load grows, even before the message consumption starts falling behind I am seeing a flood of the "duplicate" messages. They of course are not a problem by themselves but the DLQ fills up really quickly and AMQs die horrible out of memory death. Exactly the same behavior under the same setup and load is observed for the 5 or .11-SNAPSHOT (I basically swap the activemq link in the /opt folder)
6) Keeping the whole system the same, when I switch to 5.9.0 the issue disappears - no "duplicate" in the log, DLQ does not get populated.

So the question is what gives? The consumers seem to process all the messages at least it shouldn't be different from 5.9 to 5.10 to 5.11. Do you believe something has changed in 5.10 and it might be a legit behavior?

Let me know. I can provide additional details and assist with debugging if needed

Thank you,

Alex

)

> "cursor got duplicate" error after upgrade
> ------------------------------------------
>
>                 Key: AMQ-5249
>                 URL: https://issues.apache.org/jira/browse/AMQ-5249
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.9.1, 5.10.0
>            Reporter: Rural Hunter
>
> I was using 5.9.0 and meet one problem so I tried to upgrade activemq. I tried both 5.9.1 and 5.10.0 and encouterred a same problem. I saw messages filled DLQ very quickly. I checked the clients both producer and consumer but there was no error. I checked activemq log and found the log is full of these warnings:
> 2014-06-27 23:22:09,337 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.cyyun.webmon.spider.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:211.com-52399-1400732399425-1:1:235992:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,337 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:nbzjjf22805-34129-1403880308671-1:1:28:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:jxncxnj2-48598-1403856107346-1:1:6007:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:jxnc17-60227-1400730816361-1:1:149072:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:cyyun-46954-1403800808565-1:1:9765:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:ubuntu-55495-1403497638437-1:1:53086:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,340 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true - cursor got duplicate: ID:cyyun-39030-1403880008363-1:1:70:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
> The problem mostly happens right after activemq starts and sometimes happened after activemq worked normally for a while.
> For now I have to roll back to 5.9.0 and the problem doesn't occure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)