You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org> on 2009/11/11 15:01:39 UTC
[jira] Created: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
UIMA AS worker does not respond to client initialize after failover
-------------------------------------------------------------------
Key: UIMA-1658
URL: https://issues.apache.org/jira/browse/UIMA-1658
Project: UIMA
Issue Type: Bug
Components: Async Scaleout
Reporter: Jörn Kottmann
Fix For: 2.3AS
A Pure Master Slave Broker is used to increase availability of the broker.
More information about it can be found in the activemq documentation:
http://activemq.apache.org/pure-master-slave.html
In a test we simulated Master failure through killing the process with kill -9.
Here is the log output from the worker node:
INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
##### Here I stopped the master broker process with kill -9 #####
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
Afterwards the client was restarted but got a time out error during initialize.
During initializes it sends a message to the worker nodes input queue, but this
message if never retrieved.
I used the activemq web interface to get some information about the message:
Command 2001
MessageFrom ID:XXXX-51032-1257865414664-0:1:1
ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jerry Cwiklik (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777038#action_12777038 ]
Jerry Cwiklik commented on UIMA-1658:
-------------------------------------
Jorn, I was able to recreate the scenario. When the primary broker is killed, I can see in jConsole that the UIMA AS service silently reconnects to the remaining broker. The number of consumers is incremented as expected. I can also see that the client sends GetMeta request to the queue in the remaining broker. The enqueue count is incremented, however the dispatch count and dequeue count are not incremented. This suggests that there is some problem either in AMQ or Spring. There is no indication in the UIMA AS service that there is a problem. No exception is thrown from Spring when the primary broker is killed (as expected). The getMeta request is not being delivered to the UIMA AS listener object. Can you confirm that this is happening with you as well?
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Jörn Kottmann
> Fix For: 2.3AS
>
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann updated UIMA-1658:
--------------------------------
Affects Version/s: 2.3AS
defer past 2.3.0
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3AS
> Environment: Ubuntu 8.10 Server, Java 1.6 and ActiveMQ 5.3.0
> Reporter: Jörn Kottmann
> Fix For: 2.3AS
>
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann updated UIMA-1658:
--------------------------------
Description:
A Pure Master Slave Broker is used to increase availability of the broker.
More information about it can be found in the activemq documentation:
http://activemq.apache.org/pure-master-slave.html
In a test we simulated Master failure through killing the process with kill -9.
Here is the log output from the worker node:
INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
Here I stopped the master broker process with kill -9
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
Afterwards the client was restarted but got a time out error during initialize.
During initializes it sends a message to the worker nodes input queue, but this
message if never retrieved.
I used the activemq web interface to get some information about the message:
Command 2001
MessageFrom ID:XXXX-51032-1257865414664-0:1:1
ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
MessageType 3000
was:
A Pure Master Slave Broker is used to increase availability of the broker.
More information about it can be found in the activemq documentation:
http://activemq.apache.org/pure-master-slave.html
In a test we simulated Master failure through killing the process with kill -9.
Here is the log output from the worker node:
INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
##### Here I stopped the master broker process with kill -9 #####
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
Afterwards the client was restarted but got a time out error during initialize.
During initializes it sends a message to the worker nodes input queue, but this
message if never retrieved.
I used the activemq web interface to get some information about the message:
Command 2001
MessageFrom ID:XXXX-51032-1257865414664-0:1:1
ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
MessageType 3000
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Jörn Kottmann
> Fix For: 2.3AS
>
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann updated UIMA-1658:
--------------------------------
Fix Version/s: (was: 2.3AS)
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3AS
> Environment: Ubuntu 8.10 Server, Java 1.6 and ActiveMQ 5.3.0
> Reporter: Jörn Kottmann
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777041#action_12777041 ]
Jörn Kottmann commented on UIMA-1658:
-------------------------------------
Yes, I think the same thing is happening on our side too. I used a remote debugger to have a look at the stack of the threads on the worker node and it looked like that two threads are waiting to receive a message inside activemq.
I tried to add the timeout=3000 parameter to the broker uri, but it seems not to help.
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Jörn Kottmann
> Fix For: 2.3AS
>
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1658) UIMA AS worker does not respond to
client initialize after failover
Posted by "Jörn Kottmann (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörn Kottmann updated UIMA-1658:
--------------------------------
Environment: Ubuntu 8.10 Server, Java 1.6 and ActiveMQ 5.3.0
> UIMA AS worker does not respond to client initialize after failover
> -------------------------------------------------------------------
>
> Key: UIMA-1658
> URL: https://issues.apache.org/jira/browse/UIMA-1658
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Environment: Ubuntu 8.10 Server, Java 1.6 and ActiveMQ 5.3.0
> Reporter: Jörn Kottmann
> Fix For: 2.3AS
>
>
> A Pure Master Slave Broker is used to increase availability of the broker.
> More information about it can be found in the activemq documentation:
> http://activemq.apache.org/pure-master-slave.html
> In a test we simulated Master failure through killing the process with kill -9.
> Here is the log output from the worker node:
> INFO FailoverTransport - Successfully connected to tcp://XXX1:61616
> Here I stopped the master broker process with kill -9
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> WARN FailoverTransport - Transport failed to tcp://XXX1:61616 , attempting to automatically reconnect due to: java.io.EOFException
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> INFO FailoverTransport - Successfully reconnected to tcp://XXX2:61616
> Afterwards the client was restarted but got a time out error during initialize.
> During initializes it sends a message to the worker nodes input queue, but this
> message if never retrieved.
> I used the activemq web interface to get some information about the message:
> Command 2001
> MessageFrom ID:XXXX-51032-1257865414664-0:1:1
> ServerURI failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false
> MessageType 3000
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.