You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Dariusz Szablinski (JIRA)" <ji...@apache.org> on 2010/03/10 15:45:48 UTC

[jira] Created: (AMQ-2645) Deadlock during FailoverTransport interruption

Deadlock during FailoverTransport interruption
----------------------------------------------

                 Key: AMQ-2645
                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.3.1
            Reporter: Dariusz Szablinski
            Priority: Blocker
             Fix For: 5.3.1
         Attachments: screenshot-1.JPG, screenshot-2.JPG

Here is my scenario:
1)	Two AMQ servers setup with shared persistence storage (replicated message store)
2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".

During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).

This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Dariusz Szablinski (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58315#action_58315 ] 

Dariusz Szablinski commented on AMQ-2645:
-----------------------------------------

Thx. Any chance this fix will appear also in 5.3.1?

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.4.0
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rob Davies resolved AMQ-2645.
-----------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 5.3.1)
                   5.4.0

Fixed by svn revision 924350

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.4.0
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58142#action_58142 ] 

Rob Davies commented on AMQ-2645:
---------------------------------

Thanks for raising this - can you attach the screenshots ? - would help in quickly identifying the issue

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Priority: Blocker
>             Fix For: 5.3.1
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Dariusz Szablinski (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dariusz Szablinski updated AMQ-2645:
------------------------------------

    Attachment: screenshot-1.JPG

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Priority: Blocker
>             Fix For: 5.3.1
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rob Davies reassigned AMQ-2645:
-------------------------------

    Assignee: Rob Davies

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.3.1
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Dariusz Szablinski (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dariusz Szablinski updated AMQ-2645:
------------------------------------

    Attachment: screenshot-2.JPG

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Priority: Blocker
>             Fix For: 5.3.1
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AMQ-2645) Deadlock during FailoverTransport interruption

Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/activemq/browse/AMQ-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58374#action_58374 ] 

Rob Davies commented on AMQ-2645:
---------------------------------

There's changes to the transport interface to support this - so its not possible to put into 5.3.1

> Deadlock during FailoverTransport interruption
> ----------------------------------------------
>
>                 Key: AMQ-2645
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2645
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.1
>            Reporter: Dariusz Szablinski
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.4.0
>
>         Attachments: screenshot-1.JPG, screenshot-2.JPG
>
>
> Here is my scenario:
> 1)	Two AMQ servers setup with shared persistence storage (replicated message store)
> 2)	Single client constantly sends/receives small messages to/form one queue and uses single Connection with failover transport configured. The client code you can find here: AMQ-2512.
> 3)	After pushing several messages, shutdown active AMQ node - second one will kick in; client will successfully reconnect to this new active node; however it won't be able to receive any message out of the queue. Following warning is also generated by the client every 15s: "WARNING: dispatch paused, waiting for outstanding dispatch interruption processing (20) to complete..".
> During debugging I've found that code which is supposed to clean messages during reconnection await for MessageDispatchChannel mutex owned by one of the session threads (screenshot-1). This session thread, from the other hand, is in the middle of send operation and waits for server's response. In result messages dispatch awaits infinitely on CountDownLatch which is supposed to be decremented by the deadlocked messages cleaning code (screenshot-2).
> This occurs in the latest 5.3.1-SNAPSHOT as well as in 5.3.0-fuse-01-00 version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.