You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org> on 2010/12/23 11:44:03 UTC
[jira] Created: (AMQ-3103) Active MQ stalls after Job Scheduler
component shuts down.
Active MQ stalls after Job Scheduler component shuts down.
----------------------------------------------------------
Key: AMQ-3103
URL: https://issues.apache.org/jira/browse/AMQ-3103
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.2
Environment: Redhat Enterprise Linux 5.X,
JDK 1.5 32-bit
JDK 1.6 64-bit
Active MQ 4.2
Reporter: Swapnonil Mukherjee
Observation
----
Active MQ stops accepting all incoming messages destined for a particular, queue after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{noformat}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Active MQ stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Description:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{code:xml}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
was:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular, queue after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{noformat}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
> Active MQ stalls after Job Scheduler component shuts down.
> ----------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Active MQ 4.2
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {code:xml}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {noformat}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Description:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
The only way to recover is to completely delete the data directory.
I am attaching the activemq log and activemq configuration file.
was:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
The only way to recover is the completely delete the data directory.
I am attaching the activemq log and activemq configuration file.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005116#comment-13005116 ]
Swapnonil Mukherjee commented on AMQ-3103:
------------------------------------------
Hi Timothy
I will test against 5.5.0 Snapshot and post my observations.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swen Moczarski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989816#comment-12989816 ]
Swen Moczarski commented on AMQ-3103:
-------------------------------------
We got similar problems. The scheduler stops from time to time due to an exception. Maybe, the cause is an unsynchronized method in the scheduler which results in a broken store structure. I described the issue in AMQ-3140. Testcase and patch is available :-) Would be interesting to see if the patch fix this issue.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Summary: Queue stalls after Job Scheduler component shuts down. (was: Active MQ Queue stalls after the Job Scheduler component shuts down.)
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Priority: Critical (was: Major)
Raising this to critical as this bug is causing *loss of messages*. The only way to recover from this error is to delete the data directory completely.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982055#action_12982055 ]
Swapnonil Mukherjee commented on AMQ-3103:
------------------------------------------
The broker shutdown yesterday *at the same time i.e 01:49 hours GMT on the 14th of January 2010*. The broker ran ok for about 20 days.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975058#action_12975058 ]
Swapnonil Mukherjee commented on AMQ-3103:
------------------------------------------
We deleted the data directory, and restarted the broker. After this the scheduler error has not re-appeared.
So the conclusion is that if the Job Scheduler component encounters an error as listed above, then just deleting the db.redo and schedulerDB.redo file is not enough. We needed to delete the entire data directory.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Active MQ Queue stalls after the Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974842#action_12974842 ]
Swapnonil Mukherjee commented on AMQ-3103:
------------------------------------------
Hi Everybody,
Another observation. This is one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another activemq.log file. This one is from our staging servers.
> Active MQ Queue stalls after the Job Scheduler component shuts down.
> --------------------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-3103) Queue stalls after Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974842#action_12974842 ]
Swapnonil Mukherjee edited comment on AMQ-3103 at 12/24/10 5:28 AM:
--------------------------------------------------------------------
Hi Everybody,
Another observation. This one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another log file named activemq-1.log. This one is from our staging servers where you would observe that the Job Scheduler shuts itself precisely at 01:48 hours.
was (Author: swapnonil):
Hi Everybody,
Another observation. This is one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another log file named activemq-1.log. This one is from our staging servers where you would observe that the Job Scheduler shuts itself precisely at 01:48 hours.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Active MQ Queue stalls after the Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Summary: Active MQ Queue stalls after the Job Scheduler component shuts down. (was: Active MQ stalls after Job Scheduler component shuts down.)
> Active MQ Queue stalls after the Job Scheduler component shuts down.
> --------------------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-3103) Queue stalls after Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974842#action_12974842 ]
Swapnonil Mukherjee edited comment on AMQ-3103 at 12/24/10 12:48 AM:
---------------------------------------------------------------------
Hi Everybody,
Another observation. This is one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another log file named activemq-1.log. This one is from our staging servers where you would observe that the Job Scheduler shuts itself precisely at 01:48 hours.
was (Author: swapnonil):
Hi Everybody,
Another observation. This is one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another activemq.log file. This one is from our staging servers.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Active MQ stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Description:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
was:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{code:xml}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
> Active MQ stalls after Job Scheduler component shuts down.
> ----------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Active MQ 4.2
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Active MQ stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Environment:
Redhat Enterprise Linux 5.X,
JDK 1.5 32-bit
JDK 1.6 64-bit
was:
Redhat Enterprise Linux 5.X,
JDK 1.5 32-bit
JDK 1.6 64-bit
Active MQ 4.2
> Active MQ stalls after Job Scheduler component shuts down.
> ----------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004871#comment-13004871 ]
Timothy Bish commented on AMQ-3103:
-----------------------------------
There have been several fixes to the scheduler in trunk, have you tested against a 5.5.0 SNAPSHOT to see if this has been resolved?
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (AMQ-3103) Queue stalls after Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982055#action_12982055 ]
Swapnonil Mukherjee edited comment on AMQ-3103 at 1/15/11 5:08 AM:
-------------------------------------------------------------------
The broker shutdown yesterday *at the same time i.e 01:49 hours GMT on the 14th of January 2011*. The broker ran ok for about 20 days.
was (Author: swapnonil):
The broker shutdown yesterday *at the same time i.e 01:49 hours GMT on the 14th of January 2010*. The broker ran ok for about 20 days.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Description:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
The only way to recover is the completely delete the data directory.
I am attaching the activemq log and activemq configuration file.
was:
Observation
----
Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
{code:xml}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{code}
Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
I am attaching the activemq log and activemq configuration file.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is the completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Laslo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988793#comment-12988793 ]
Laslo commented on AMQ-3103:
----------------------------
Hello,
we got the same bug in our setup. The differenz is that there is no observable reason why the scheduler throws the exception and shut down.
Is somebody working on this issue?
Thanks
Laslo
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-3103) Active MQ Queue stalls after the Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Attachment: activemq-1.log
> Active MQ Queue stalls after the Job Scheduler component shuts down.
> --------------------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036228#comment-13036228 ]
Swapnonil Mukherjee commented on AMQ-3103:
------------------------------------------
Hi,
I have not had this issue after upgrading to the Active MQ 5.5.0 Snapshot. Will upgrade all our production servers to Active Mq 5.5.0 in that case.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Comment: was deleted
(was: We deleted the data directory, and restarted the broker. After this the scheduler error has not re-appeared.
So the conclusion is that if the Job Scheduler component encounters an error as listed above, then just deleting the db.redo and schedulerDB.redo file is not enough. We needed to delete the entire data directory.)
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is the completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Closed] (AMQ-3103) Queue stalls after Job Scheduler
component shuts down.
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3103.
-----------------------------
Resolution: Fixed
Fix Version/s: 5.5.0
User reports problems went away by upgrading to a 5.5.0 build.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is to completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (AMQ-3103) Queue stalls after Job
Scheduler component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974842#action_12974842 ]
Swapnonil Mukherjee edited comment on AMQ-3103 at 1/15/11 5:25 AM:
-------------------------------------------------------------------
Hi Everybody,
Another observation. This one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the entire data directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another log file named activemq-1.log. This one is from our staging servers where you would observe that the Job Scheduler shuts itself precisely at 01:48 hours.
was (Author: swapnonil):
Hi Everybody,
Another observation. This one is funny.
We are observing that the Job Scheduler component *shuts down everyday precisely at 01:48 hours GMT.*
As mentioned earlier, the way we recover from these failures is by
* Stopping the broker using $> ./activemq stop
* Deleting the db.redo file from the kahadb directory
* Restarting the broker using $> ./activemq start
Are we missing doing something for the recovery?
I am re-attaching another log file named activemq-1.log. This one is from our staging servers where you would observe that the Job Scheduler shuts itself precisely at 01:48 hours.
> Queue stalls after Job Scheduler component shuts down.
> ------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Reporter: Swapnonil Mukherjee
> Priority: Critical
> Attachments: activemq-1.log, activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular queue, after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {code:xml}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {code}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> The only way to recover is the completely delete the data directory.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3103) Active MQ stalls after Job Scheduler
component shuts down.
Posted by "Swapnonil Mukherjee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swapnonil Mukherjee updated AMQ-3103:
-------------------------------------
Attachment: activemq.xml
activemq.log
Active MQ Log and Config files from our production servers.
> Active MQ stalls after Job Scheduler component shuts down.
> ----------------------------------------------------------
>
> Key: AMQ-3103
> URL: https://issues.apache.org/jira/browse/AMQ-3103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Redhat Enterprise Linux 5.X,
> JDK 1.5 32-bit
> JDK 1.6 64-bit
> Active MQ 4.2
> Reporter: Swapnonil Mukherjee
> Attachments: activemq.log, activemq.xml
>
>
> Observation
> ----
> Active MQ stops accepting all incoming messages destined for a particular, queue after the scheduler component processing scheduled messages on that queue encounters an Null Pointer Exception.
> Environment
> ----
> We are using the Spring JMSTemplate component to post messages onto a queue. We also place a delay of 30 seconds on each message before posting
> {noformat}
> message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, Integer.parseInt("30") * 1000);
> {noformat}
> We use the Spring Default Message Listener Container to receive messages.
> Normally the broker runs fine and we have seen messages appear under the "Scheduled" tab on the Active MQ Console, after which they processed normally and we can tally using the "Messages Enqueued" and the "Messages Dequeued" numbers. But occasionally the Job Scheduler fails with the following exception.
> {noformat}
> 2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job | org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.NullPointerException
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
> at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
> at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
> at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:619)
> 2010-12-10 16:31:39,561 | INFO | JobSchedulerStore:activemq-data/primary/scheduler stopped | org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
> {noformat}
> Why does the Job Scheduler fail? One possible reason we have found is that the clock time settings on the VMs producing the messages and the broker as well as the consumers are all different. So the Job Scheduler may be shutting itself down arbitrarily due to this difference in clock. We are in the process of syncing all clocks but we are not sure whether this will solve the problem.
> Bug
> ----
> But the Bug really is, even if the Job Scheduler encounters an Null Pointer, why should it shutdown? Even more problematic is the fact that the queue itself stalls and does not accept anymore messages after the Job Scheduler shuts down.
> We have tried to delete the db.redo log to recover from this type of shutdown. The broker recovers fine, but all messages posted to this queue after the Job Scheduler shut itself down were lost. We have not been able to recover those messages.
> I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.