You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Bill Wan (JIRA)" <ji...@apache.org> on 2012/11/12 09:36:14 UTC

[jira] [Created] (AMQ-4170) Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page

Bill Wan created AMQ-4170:
-----------------------------

             Summary: Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page 
                 Key: AMQ-4170
                 URL: https://issues.apache.org/jira/browse/AMQ-4170
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.5.1
            Reporter: Bill Wan


I add below configuration to clean inactive destination and expired message for every 2 hours

<destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

After MQ run for several days, it will crash when scheduler do such clean job, the crash exception is:

2012-11-12 01:03:58,917 | INFO  | 5099ae91f17021cb20b0b157 Inactive for longer than 7200000 ms - removing ... | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[broker-cluster-194] Scheduler
2012-11-12 01:04:01,907 | INFO  | Transport failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///183.177.33.93:39617
2012-11-12 01:04:02,288 | ERROR | KahaDB failed to store to Journal | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Broker[broker-cluster-194] Scheduler
java.io.EOFException: Chunk stream does not exist at page: 1868
	at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:454)
	at org.apache.kahadb.page.Transaction$2.<init>(Transaction.java:431)
	at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)
	at org.apache.kahadb.page.Transaction.load(Transaction.java:404)
	at org.apache.kahadb.page.Transaction.load(Transaction.java:361)
	at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
	at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
	at org.apache.kahadb.index.BTreeIndex.clear(BTreeIndex.java:202)
	at org.apache.activemq.store.kahadb.MessageDatabase$MessageOrderIndex.remove(MessageDatabase.java:2177)
	at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1105)
	at org.apache.activemq.store.kahadb.MessageDatabase$16.execute(MessageDatabase.java:936)
	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:934)
	at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:884)
	at org.apache.activemq.store.kahadb.data.KahaRemoveDestinationCommand.visit(KahaRemoveDestinationCommand.java:130)
	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:856)
	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:783)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAllMessages(KahaDBStore.java:384)
	at org.apache.activemq.store.ProxyMessageStore.removeAllMessages(ProxyMessageStore.java:56)
	at org.apache.activemq.broker.region.BaseDestination.dispose(BaseDestination.java:503)
	at org.apache.activemq.broker.region.AbstractRegion.dispose(AbstractRegion.java:511)
	at org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:198)
	at org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62)
	at org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:324)
	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
	at org.apache.activemq.advisory.AdvisoryBroker.removeDestination(AdvisoryBroker.java:173)
	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
	at org.apache.activemq.broker.MutableBrokerFilter.removeDestination(MutableBrokerFilter.java:155)
	at org.apache.activemq.broker.region.RegionBroker.purgeInactiveDestinations(RegionBroker.java:959)
	at org.apache.activemq.broker.region.RegionBroker$1.run(RegionBroker.java:110)
	at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AMQ-4170) Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page

Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506852#comment-13506852 ] 

Timothy Bish commented on AMQ-4170:
-----------------------------------

Do you have a test case that reproduce this?
Have you tried with a newer broker version?  There's been a lot of work done since v5.5.1 to harden KahaDB.
                
> Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page 
> -------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-4170
>                 URL: https://issues.apache.org/jira/browse/AMQ-4170
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.5.1
>            Reporter: Bill Wan
>         Attachments: activemq.log
>
>
> I add below configuration to clean inactive destination and expired message for every 2 hours
> <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
>                   <pendingSubscriberPolicy>
>                     <vmCursor />
>                   </pendingSubscriberPolicy>
>                 </policyEntry>
>                 <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
>                   <pendingQueuePolicy>
>                     <vmQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
> After MQ run for several days, it will crash when scheduler do such clean job, the crash exception is:
> 2012-11-12 01:03:58,917 | INFO  | 5099ae91f17021cb20b0b157 Inactive for longer than 7200000 ms - removing ... | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[broker-cluster-194] Scheduler
> 2012-11-12 01:04:01,907 | INFO  | Transport failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///183.177.33.93:39617
> 2012-11-12 01:04:02,288 | ERROR | KahaDB failed to store to Journal | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Broker[broker-cluster-194] Scheduler
> java.io.EOFException: Chunk stream does not exist at page: 1868
> 	at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:454)
> 	at org.apache.kahadb.page.Transaction$2.<init>(Transaction.java:431)
> 	at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)
> 	at org.apache.kahadb.page.Transaction.load(Transaction.java:404)
> 	at org.apache.kahadb.page.Transaction.load(Transaction.java:361)
> 	at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> 	at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> 	at org.apache.kahadb.index.BTreeIndex.clear(BTreeIndex.java:202)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$MessageOrderIndex.remove(MessageDatabase.java:2177)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1105)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$16.execute(MessageDatabase.java:936)
> 	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:934)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:884)
> 	at org.apache.activemq.store.kahadb.data.KahaRemoveDestinationCommand.visit(KahaRemoveDestinationCommand.java:130)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:856)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:783)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAllMessages(KahaDBStore.java:384)
> 	at org.apache.activemq.store.ProxyMessageStore.removeAllMessages(ProxyMessageStore.java:56)
> 	at org.apache.activemq.broker.region.BaseDestination.dispose(BaseDestination.java:503)
> 	at org.apache.activemq.broker.region.AbstractRegion.dispose(AbstractRegion.java:511)
> 	at org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:198)
> 	at org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62)
> 	at org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:324)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.advisory.AdvisoryBroker.removeDestination(AdvisoryBroker.java:173)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.MutableBrokerFilter.removeDestination(MutableBrokerFilter.java:155)
> 	at org.apache.activemq.broker.region.RegionBroker.purgeInactiveDestinations(RegionBroker.java:959)
> 	at org.apache.activemq.broker.region.RegionBroker$1.run(RegionBroker.java:110)
> 	at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AMQ-4170) Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page

Posted by "Bill Wan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AMQ-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bill Wan updated AMQ-4170:
--------------------------

    Attachment: activemq.log

add exception log
                
> Crash when clean inactive destination: java.io.EOFException: Chunk stream does not exist at page 
> -------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-4170
>                 URL: https://issues.apache.org/jira/browse/AMQ-4170
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.5.1
>            Reporter: Bill Wan
>         Attachments: activemq.log
>
>
> I add below configuration to clean inactive destination and expired message for every 2 hours
> <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
>                   <pendingSubscriberPolicy>
>                     <vmCursor />
>                   </pendingSubscriberPolicy>
>                 </policyEntry>
>                 <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="7200000" expireMessagesPeriod="7200000">
>                   <pendingQueuePolicy>
>                     <vmQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
> After MQ run for several days, it will crash when scheduler do such clean job, the crash exception is:
> 2012-11-12 01:03:58,917 | INFO  | 5099ae91f17021cb20b0b157 Inactive for longer than 7200000 ms - removing ... | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[broker-cluster-194] Scheduler
> 2012-11-12 01:04:01,907 | INFO  | Transport failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///183.177.33.93:39617
> 2012-11-12 01:04:02,288 | ERROR | KahaDB failed to store to Journal | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Broker[broker-cluster-194] Scheduler
> java.io.EOFException: Chunk stream does not exist at page: 1868
> 	at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:454)
> 	at org.apache.kahadb.page.Transaction$2.<init>(Transaction.java:431)
> 	at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)
> 	at org.apache.kahadb.page.Transaction.load(Transaction.java:404)
> 	at org.apache.kahadb.page.Transaction.load(Transaction.java:361)
> 	at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> 	at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> 	at org.apache.kahadb.index.BTreeIndex.clear(BTreeIndex.java:202)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$MessageOrderIndex.remove(MessageDatabase.java:2177)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1105)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$16.execute(MessageDatabase.java:936)
> 	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:934)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:884)
> 	at org.apache.activemq.store.kahadb.data.KahaRemoveDestinationCommand.visit(KahaRemoveDestinationCommand.java:130)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:856)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:783)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAllMessages(KahaDBStore.java:384)
> 	at org.apache.activemq.store.ProxyMessageStore.removeAllMessages(ProxyMessageStore.java:56)
> 	at org.apache.activemq.broker.region.BaseDestination.dispose(BaseDestination.java:503)
> 	at org.apache.activemq.broker.region.AbstractRegion.dispose(AbstractRegion.java:511)
> 	at org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:198)
> 	at org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62)
> 	at org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:324)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.advisory.AdvisoryBroker.removeDestination(AdvisoryBroker.java:173)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:149)
> 	at org.apache.activemq.broker.MutableBrokerFilter.removeDestination(MutableBrokerFilter.java:155)
> 	at org.apache.activemq.broker.region.RegionBroker.purgeInactiveDestinations(RegionBroker.java:959)
> 	at org.apache.activemq.broker.region.RegionBroker$1.run(RegionBroker.java:110)
> 	at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira