You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Michael Black (JIRA)" <ji...@apache.org> on 2011/09/16 14:52:08 UTC

[jira] [Created] (AMQ-3503) Too many open files for db log

Too many open files for db log
------------------------------

                 Key: AMQ-3503
                 URL: https://issues.apache.org/jira/browse/AMQ-3503
             Project: ActiveMQ
          Issue Type: Bug
          Components: Message Store
    Affects Versions: 5.5.0
         Environment: Redhat 5.7.  Default conf file used for startup
            Reporter: Michael Black
            Priority: Blocker


lsof | grep apache | grep data/localhost | wc -l
Shows constantly increasing number of open files. Had 1016 open when it died.
ulimit is set at 1024.
Docs say files are supposed to be removed when "no longer needed".
We're running 3 producers and 3 consumers so no messages should be left in the queue.
At the point of dying we have put in around 2 billion messages.

2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
  at java.io.RandomAccessFile.open(Native Method)
  at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
  at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
  at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
  at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
  at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
  at org.apache.kahadb.journal.Journal.read(Journal.java:598)
  at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
  at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
  at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
  at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
  at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
  at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
  at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
  at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AMQ-3503) Too many open files for db log

Posted by "Michael Black (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107785#comment-13107785 ] 

Michael Black commented on AMQ-3503:
------------------------------------

OK...I think we can close this now as of the latest snapshot
I've pumped in about 5 billion message.
I've got 207 files open now so it's not increasing at all.

> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AMQ-3503) Too many open files for db log

Posted by "Michael Black (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106092#comment-13106092 ] 

Michael Black commented on AMQ-3503:
------------------------------------

The db log files seem to be getting deleted now...whereas before (in 5.5.0) all the db-*-.log files were still in the directory (over 1000 of them).

So...for right now the latest snapshot seems to have fixed this.

I'll follow up later after a few billion messages have been put in the queue and it's still running.

That will be in a day or two.

Total files open by apachemq seems stable at around 212-215

lsof | grep apache | wc -l



ls -l
total 96360
-rw-rw-r-- 1 root root 33554649 Sep 16 10:21 db-44.log
-rw-rw-r-- 1 root root 33554465 Sep 16 10:21 db-45.log
-rw-rw-r-- 1 root root 33030144 Sep 16 10:21 db-46.log
-rw-rw-r-- 1 root root        0 Sep 16 10:17 lock
-rw-rw-r-- 1 root root    12288 Sep 16 10:17 tmpDB.data
-rw-rw-r-- 1 root root    12304 Sep 16 10:17 tmpDB.redo

Michael D. Black

Senior Scientist

NG Information Systems

Advanced Analytics Directorate





> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Closed] (AMQ-3503) Too many open files for db log

Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Timothy Bish closed AMQ-3503.
-----------------------------

    Resolution: Not A Problem

PList updates have resolved this issue.

> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AMQ-3503) Too many open files for db log

Posted by "Torsten Mielke (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106065#comment-13106065 ] 

Torsten Mielke commented on AMQ-3503:
-------------------------------------

You can get the latest snapshot version from [here|https://repository.apache.org/content/groups/snapshots-group/org/apache/activemq/apache-activemq/5.6-SNAPSHOT/].

> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AMQ-3503) Too many open files for db log

Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106035#comment-13106035 ] 

Gary Tully commented on AMQ-3503:
---------------------------------

can you try this scenario with a 5.6-SNAPSHOT, the plist store has had a revamp since 5.5

> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (AMQ-3503) Too many open files for db log

Posted by "Michael Black (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106043#comment-13106043 ] 

Michael Black commented on AMQ-3503:
------------------------------------

Sure...can you tell me how?  A link somewhere?



I'm not exactly a Java wiz....



Michael D. Black

Senior Scientist

NG Information Systems

Advanced Analytics Directorate





> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor | Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log (Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira