You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Andrew MacBean (Created) (JIRA)" <ji...@apache.org> on 2011/11/25 11:48:39 UTC
[jira] [Created] (QPID-3646) Java Broker becomes unresponsive after
BDB Null Pointer Exception encountered
Java Broker becomes unresponsive after BDB Null Pointer Exception encountered
-----------------------------------------------------------------------------
Key: QPID-3646
URL: https://issues.apache.org/jira/browse/QPID-3646
Project: Qpid
Issue Type: Bug
Components: Java Broker, Java Broker BDB Store, Java Client
Affects Versions: 0.5
Reporter: Andrew MacBean
Java Broker becomes unresponsive after BDB Null Pointer Exception encountered in the logs. System encountered disk space issues previously but these were not thought to have affected BDB, only QPID logs. The BDBStore was stored on a different file system that was healthy.
The error was observed and then later on clients had issues connecting to the broker and it was required to be restarted. The NPE seemed to be thrown by a BDB cleaner thread that had an invalid offset.
The BDB Store is version: 4.0.103
Recorded Timeline of Events
1) Nov 23 12:40:57
The filesystem where QPID code resides and the logs are written gets filled up , due to another process, and a message is observed in QPID log:
log4j:ERROR Failed to flog4j:ERROR Failed to flush writer,
java.io.IOException: No space left on device
Disk space issue was then resolved by 1300.
2) Nov 23 15:27:41
The following message appears in the QPID log:
<DaemonThread name="Cleaner-1"/> caught exception: java.lang.NullPointerException
java.lang.NullPointerException at com.sleepycat.je.cleaner.OffsetList$Segment.get(OffsetList.java:192)
at com.sleepycat.je.cleaner.OffsetList.contains(OffsetList.java:135)
at com.sleepycat.je.cleaner.TrackedFileSummary.containsObsoleteOffset(TrackedFileSummary.java:169)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:479)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:241)
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:140)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:160)
at java.lang.Thread.run(Thread.java:662)
3) Around Nov 23 16:30
Clients start reporting QPID is not responding and report the following errors:
org.apache.qpid.client.JMSAMQException: Failed to commit: Server did not respond in a timely fashion [error code 408: Request Timeout]
and
Error creating session: org.apache.qpid.AMQTimeoutException: Server did not respond in a timely fashion [error code 408: Request Timeout]
Errors in qpid log seen:
2011-11-23 16:29:12,715 ERROR [SocketAcceptorIoProcessor-0.3] protocol.AMQPFastProtocolHandler (AMQPFastProtocolHandler.java:223) - IOException caught in/169.93.5.123:56138(notifier_ods), session closed implictly: java.io.IOException: Connection reset by peer
and
2011-11-23 16:33:41,495 WARN [Queue-housekeeping-notifier] server.AMQChannel (AMQChannel.java:1071) - IDLE TRANSACTION ALERT [con:1,069(notifier_ods@/169.93.5.123:55879/notifier)/ch:3] 270932 ms
QPID Management Console is frozen showing no updates
QPID Jconsole doesn't show anything wrong
4) Nov 23 17:15:26
QPID is bounced and no further issues are observed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Resolved] (QPID-3646) Java Broker becomes unresponsive
after BDB Null Pointer Exception encountered
Posted by "Robbie Gemmell (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robbie Gemmell resolved QPID-3646.
----------------------------------
Resolution: Not A Problem
Resolving this, as per the Oracle forum there is little we can do about the BDB NPE except put it down to magic, and I believe there were additional circumstances we later became aware of around the unresponsiveness of that broker that mean we can be assured it had nothing to do with the NPE.
> Java Broker becomes unresponsive after BDB Null Pointer Exception encountered
> -----------------------------------------------------------------------------
>
> Key: QPID-3646
> URL: https://issues.apache.org/jira/browse/QPID-3646
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker, Java Broker BDB Store, Java Client
> Affects Versions: 0.5
> Reporter: Andrew MacBean
>
> Java Broker becomes unresponsive after BDB Null Pointer Exception encountered in the logs. System encountered disk space issues previously but these were not thought to have affected BDB, only QPID logs. The BDBStore was stored on a different file system that was healthy.
> The error was observed and then later on clients had issues connecting to the broker and it was required to be restarted. The NPE seemed to be thrown by a BDB cleaner thread that had an invalid offset.
> The BDB Store is version: 4.0.103
> Recorded Timeline of Events
> 1) Nov 23 12:40:57
> The filesystem where QPID code resides and the logs are written gets filled up , due to another process, and a message is observed in QPID log:
> log4j:ERROR Failed to flog4j:ERROR Failed to flush writer,
> java.io.IOException: No space left on device
> Disk space issue was then resolved by 1300.
> 2) Nov 23 15:27:41
> The following message appears in the QPID log:
> <DaemonThread name="Cleaner-1"/> caught exception: java.lang.NullPointerException
> java.lang.NullPointerException at com.sleepycat.je.cleaner.OffsetList$Segment.get(OffsetList.java:192)
> at com.sleepycat.je.cleaner.OffsetList.contains(OffsetList.java:135)
> at com.sleepycat.je.cleaner.TrackedFileSummary.containsObsoleteOffset(TrackedFileSummary.java:169)
> at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:479)
> at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:241)
> at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:140)
> at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:160)
> at java.lang.Thread.run(Thread.java:662)
> 3) Around Nov 23 16:30
> Clients start reporting QPID is not responding and report the following errors:
> org.apache.qpid.client.JMSAMQException: Failed to commit: Server did not respond in a timely fashion [error code 408: Request Timeout]
> and
> Error creating session: org.apache.qpid.AMQTimeoutException: Server did not respond in a timely fashion [error code 408: Request Timeout]
> Errors in qpid log seen:
> 2011-11-23 16:29:12,715 ERROR [SocketAcceptorIoProcessor-0.3] protocol.AMQPFastProtocolHandler (AMQPFastProtocolHandler.java:223) - IOException caught in/169.93.5.123:56138(notifier_ods), session closed implictly: java.io.IOException: Connection reset by peer
> and
> 2011-11-23 16:33:41,495 WARN [Queue-housekeeping-notifier] server.AMQChannel (AMQChannel.java:1071) - IDLE TRANSACTION ALERT [con:1,069(notifier_ods@/169.93.5.123:55879/notifier)/ch:3] 270932 ms
> QPID Management Console is frozen showing no updates
> QPID Jconsole doesn't show anything wrong
> 4) Nov 23 17:15:26
> QPID is bounced and no further issues are observed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Commented] (QPID-3646) Java Broker becomes unresponsive
after BDB Null Pointer Exception encountered
Posted by "Andrew MacBean (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157096#comment-13157096 ]
Andrew MacBean commented on QPID-3646:
--------------------------------------
Posted on the Oracle Berkeley DB Java Edition Forum:
https://forums.oracle.com/forums/message.jspa?messageID=10002248#10002248
> Java Broker becomes unresponsive after BDB Null Pointer Exception encountered
> -----------------------------------------------------------------------------
>
> Key: QPID-3646
> URL: https://issues.apache.org/jira/browse/QPID-3646
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker, Java Broker BDB Store, Java Client
> Affects Versions: 0.5
> Reporter: Andrew MacBean
>
> Java Broker becomes unresponsive after BDB Null Pointer Exception encountered in the logs. System encountered disk space issues previously but these were not thought to have affected BDB, only QPID logs. The BDBStore was stored on a different file system that was healthy.
> The error was observed and then later on clients had issues connecting to the broker and it was required to be restarted. The NPE seemed to be thrown by a BDB cleaner thread that had an invalid offset.
> The BDB Store is version: 4.0.103
> Recorded Timeline of Events
> 1) Nov 23 12:40:57
> The filesystem where QPID code resides and the logs are written gets filled up , due to another process, and a message is observed in QPID log:
> log4j:ERROR Failed to flog4j:ERROR Failed to flush writer,
> java.io.IOException: No space left on device
> Disk space issue was then resolved by 1300.
> 2) Nov 23 15:27:41
> The following message appears in the QPID log:
> <DaemonThread name="Cleaner-1"/> caught exception: java.lang.NullPointerException
> java.lang.NullPointerException at com.sleepycat.je.cleaner.OffsetList$Segment.get(OffsetList.java:192)
> at com.sleepycat.je.cleaner.OffsetList.contains(OffsetList.java:135)
> at com.sleepycat.je.cleaner.TrackedFileSummary.containsObsoleteOffset(TrackedFileSummary.java:169)
> at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:479)
> at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:241)
> at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:140)
> at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:160)
> at java.lang.Thread.run(Thread.java:662)
> 3) Around Nov 23 16:30
> Clients start reporting QPID is not responding and report the following errors:
> org.apache.qpid.client.JMSAMQException: Failed to commit: Server did not respond in a timely fashion [error code 408: Request Timeout]
> and
> Error creating session: org.apache.qpid.AMQTimeoutException: Server did not respond in a timely fashion [error code 408: Request Timeout]
> Errors in qpid log seen:
> 2011-11-23 16:29:12,715 ERROR [SocketAcceptorIoProcessor-0.3] protocol.AMQPFastProtocolHandler (AMQPFastProtocolHandler.java:223) - IOException caught in/169.93.5.123:56138(notifier_ods), session closed implictly: java.io.IOException: Connection reset by peer
> and
> 2011-11-23 16:33:41,495 WARN [Queue-housekeeping-notifier] server.AMQChannel (AMQChannel.java:1071) - IDLE TRANSACTION ALERT [con:1,069(notifier_ods@/169.93.5.123:55879/notifier)/ch:3] 270932 ms
> QPID Management Console is frozen showing no updates
> QPID Jconsole doesn't show anything wrong
> 4) Nov 23 17:15:26
> QPID is bounced and no further issues are observed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org