You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Imesh Gunaratne (JIRA)" <ji...@apache.org> on 2015/07/09 16:04:04 UTC
[jira] [Assigned] (STRATOS-1308) Reconnection logic is missing the
connection cleaning up part
[ https://issues.apache.org/jira/browse/STRATOS-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Imesh Gunaratne reassigned STRATOS-1308:
----------------------------------------
Assignee: Imesh Gunaratne
> Reconnection logic is missing the connection cleaning up part
> -------------------------------------------------------------
>
> Key: STRATOS-1308
> URL: https://issues.apache.org/jira/browse/STRATOS-1308
> Project: Stratos
> Issue Type: Bug
> Affects Versions: 4.1.0 Alpha
> Reporter: Mariangela Hills
> Assignee: Imesh Gunaratne
> Priority: Critical
>
> I got a message saying "could not connect to the message broker" even when the message broker connection was already in place (started). There seems to be some reconnection logic, which finally leads to a too many files opened error (part of the log can be seen in [1]). When this happens the web browser on my machine (Chrome) gets stuck and I have to restart Stratos to solve the issue. In addition, I would like to add that my max file limit was 8000 when this issue occurred (ulimit -Sn 8000). Therefore, it looks like the reconnection logic is missing the connection cleaning up part.
> ---------------------
> [1]
> [2015-04-02 13:40:16,319] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 5 seconds: [topic-name] tenant.CompleteTenantEvent
> [2015-04-02 13:40:16,319] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Connection to the message broker failed
> [2015-04-02 13:40:16,319] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 2 seconds: [topic-name] tenant.CompleteTenantEvent
> Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628" java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:673)
> at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
> at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
> at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 13:40:16,423] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not connect to message broker
> javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
> at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
> at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
> at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
> at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
> at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> at org.apache.activemq.transport.ResponseCorrelator.
> request(ResponseCorrelator.java:86)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
> ... 9 more
> at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
> at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
> at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
> at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
> at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,424] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not reconnect to message broker
> org.apache.stratos.messaging.domain.exception.MessagingException: Could not connect to message broker
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
> at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
> at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
> ... 6 more
> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
> at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
> at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
> at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
> ... 9 more
> [2015-04-02 13:40:16,424] ERROR {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties (Too many open files in system)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,425] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent
> [2015-04-02 13:40:16,427] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not create topic connector
> java.lang.NullPointerException
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,427] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not reconnect to message broker
> org.apache.stratos.messaging.domain.exception.MessagingException: Could not create topic connector
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: java.lang.NullPointerException
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
> ... 6 more
> [2015-04-02 13:40:16,428] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent
> [2015-04-02 13:40:16,445] ERROR {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties (Too many open files in system)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,448] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not create topic connector
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)