You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Philip Harvey (Created) (JIRA)" <ji...@apache.org> on 2011/12/19 15:33:30 UTC

[jira] [Created] (QPID-3697) SortedQueue synchronization causes deadlock

SortedQueue synchronization causes deadlock
-------------------------------------------

                 Key: QPID-3697
                 URL: https://issues.apache.org/jira/browse/QPID-3697
             Project: Qpid
          Issue Type: Bug
          Components: Java Broker
    Affects Versions: 0.5, 0.15
         Environment: All
            Reporter: Philip Harvey


We've seen a broker become unresponsive several times in Production.

Analysis of a thread dump showed a deadlock between two threads, namely:

- a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
- a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.

The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.

Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Assigned] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Keith Wall (Assigned) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Keith Wall reassigned QPID-3697:
--------------------------------

    Assignee: Robbie Gemmell  (was: Keith Wall)
    
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Robbie Gemmell
>             Fix For: 0.15
>
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Assigned] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Keith Wall (Assigned) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Keith Wall reassigned QPID-3697:
--------------------------------

    Assignee: Keith Wall
    
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Keith Wall
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Resolved] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Robbie Gemmell (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robbie Gemmell resolved QPID-3697.
----------------------------------

    Resolution: Fixed

Change looks good to me.
                
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Robbie Gemmell
>             Fix For: 0.15
>
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Commented] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Keith Wall (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185615#comment-13185615 ] 

Keith Wall commented on QPID-3697:
----------------------------------

Patch applied.
                
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Keith Wall
>             Fix For: 0.15
>
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Updated] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Keith Wall (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Keith Wall updated QPID-3697:
-----------------------------

    Status: Ready To Review  (was: In Progress)
    
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Keith Wall
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


[jira] [Updated] (QPID-3697) SortedQueue synchronization causes deadlock

Posted by "Keith Wall (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Keith Wall updated QPID-3697:
-----------------------------

    Fix Version/s: 0.15
    
> SortedQueue synchronization causes deadlock
> -------------------------------------------
>
>                 Key: QPID-3697
>                 URL: https://issues.apache.org/jira/browse/QPID-3697
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: 0.5, 0.15
>         Environment: All
>            Reporter: Philip Harvey
>            Assignee: Keith Wall
>             Fix For: 0.15
>
>
> We've seen a broker become unresponsive several times in Production.
> Analysis of a thread dump showed a deadlock between two threads, namely:
> - a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
> - a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
> The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
> Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org