You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by alt_alt <al...@163.com> on 2016/03/09 04:56:11 UTC
Failed network connector can not be re-established
*Configuration for network connector:
*
<networkConnector name="external01_01" conduitSubscriptions="false"
uri="static://(tcp://10.102.44.181:61616)" userName="MQ-PRD-ADMIN"
password="$
{connector.password}
">
<excludedDestinations>
<topic physicalName="VirtualTopic.>"/>
</excludedDestinations>
</networkConnector>
*log:
*2016-03-03 14:41:13,070 | WARN | JDBC Failure: Borrow prepareStatement from
pool failed | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter |
ActiveMQ Transport: tcp://10.102.44.181:61616@38805
org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
pool failed
at
org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)[commons-dbcp-1.4.jar:1.4]
at
org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)[commons-dbcp-1.4.jar:1.4]
at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)[commons-dbcp-1.4.jar:1.4]
at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRemoveMessage(DefaultJDBCAdapter.java:381)[activemq-jdbc-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:260)[activemq-jdbc-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:373)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:170)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:923)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1718)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:55)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:277)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:441)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:97)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:550)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.network.DemandForwardingBridgeSupport$11.onCompletion(DemandForwardingBridgeSupport.java:1012)[activemq-broker-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:65)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.11.1.jar:5.11.1]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_80]
2016-03-03 14:41:13,073 | WARN | Transport Connection to:
vm://PRD_ACTIVEMQ_C1#0 failed: java.io.IOException: Failed to broker
message: ID:*-36573-1456931506213-1:3:1:1:7874 in container:
org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
pool failed | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp://10.102.44.181:61616@38805
2016-03-03 14:41:13,074 | INFO | PRD_ACTIVEMQ_C1 Shutting down |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[PRD_ACTIVEMQ_C1] Task-25434
2016-03-03 14:41:23,080 | INFO | Network Could not shutdown in a timely
manner | org.apache.activemq.network.DemandForwardingBridgeSupport |
ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434
2016-03-03 14:41:24,083 | INFO | PRD_ACTIVEMQ_C1 bridge to localhost stopped
| org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[PRD_ACTIVEMQ_C1] Task-25434
*It is very URGENT since we found this issue on our production environment!
Please help !!!!!!*
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by artnaseef <ar...@artnaseef.com>.
Hey Tim, the demand forwarding network connector should automatically attempt
to reconnect. At one point, years ago, I tried removing the failover
transport and sticking to the reties there because of other issues and found
that without the failover transport, the reconnects are not as reliable.
In this instance, with the exception involved, I suspect the bridge's
reconnect logic may be failing.
So, I personally always recommend using the failover transport with network
connections.
If you find otherwise, I would be interested to learn what you find.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709263.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by Tim Bain <tb...@alumni.duke.edu>.
Art, it's always been my understanding that static TCP transports would
reconnect when used as a networkConnector, and that the only reason to use
failover is if you have multiple brokers. Is that not accurate?
On Mar 12, 2016 11:17 PM, "artnaseef" <ar...@artnaseef.com> wrote:
> Ahh, here it is from the original post:
>
> uri="static://(tcp://10.102.44.181:61616)"
>
> try this instead:
>
> uri="static://(failover://(tcp://10.102.44.181:61616))"
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709242.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Failed network connector can not be re-established
Posted by alt_alt <al...@163.com>.
Thanks for you advice for tuning JDBC and configuration of network connector,
and we'll definitely try that. But it looks like the network connector is
closed from the source side due to local JDBC exception(the source side
means the source ActiveMQ server of forwarded message via network
connector), I am not sure whether failover makes difference, but we'll try
it first.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709247.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by artnaseef <ar...@artnaseef.com>.
Ahh, here it is from the original post:
uri="static://(tcp://10.102.44.181:61616)"
try this instead:
uri="static://(failover://(tcp://10.102.44.181:61616))"
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709242.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by artnaseef <ar...@artnaseef.com>.
Also - network connectors typically automatically reconnect.
Is the network connector configured to use the failover transport? If not,
try adding it to see if that helps to eliminate the problem.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709241.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by Tim Bain <tb...@alumni.duke.edu>.
Other possibilities are a bug in ActiveMQ code where we fail to close() our
PreparedStatements in certain cases, and a bug in the connection pool.
On Mar 12, 2016 11:13 PM, "artnaseef" <ar...@artnaseef.com> wrote:
> Any idea what is causing "Borrow prepareStatement from pool failed"?
>
> The only possible cause coming to mind right now is a pool that is
> exhausted
> - meaning there are long-running DB operations. If that's the case, then
> tuning the DB, moving to another store, or possible tuning the pooling
> settings for JDBC may help.
>
> It is also possible here that such a scenario will occur when there are too
> many messages backing up in the broker due to slow consumption. If so,
> then
> it is critical to eliminate the slow consumption.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709240.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Failed network connector can not be re-established
Posted by artnaseef <ar...@artnaseef.com>.
Any idea what is causing "Borrow prepareStatement from pool failed"?
The only possible cause coming to mind right now is a pool that is exhausted
- meaning there are long-running DB operations. If that's the case, then
tuning the DB, moving to another store, or possible tuning the pooling
settings for JDBC may help.
It is also possible here that such a scenario will occur when there are too
many messages backing up in the broker due to slow consumption. If so, then
it is critical to eliminate the slow consumption.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709240.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by alt_alt <al...@163.com>.
Hi Tim,
Thanks for your advice, I've already filed a jira ticket for it. It is
AMQ-6200
Add a retry mechanism would be a decent solution to my problem, and I agree
with you that just ignoring the exception is not a good one. Please add this
feature to future ActiveMQ release. We can disable pooling preparestatement.
But I am looking for another way as a workaround, I am enabling testOnBorrow
or testwhileIdle to see whether that could help since I doubt it is because
we used invalid connection to create preparestatement and throw that
exception.
Anyway, adding retry for network connector is definitely a good improvement.
:)
Best Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709217.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by Tim Bain <tb...@alumni.duke.edu>.
I think the current behavior of stopping the networkConnector when it will
be impossible for it to do any work (because we can't write to the
persistent store) is the right behavior; having a zombie networkConnector
hanging around just hoping for the store to become available sounds like an
awful option for a number of reasons, one of which is that it will "steal"
as many messages as its prefetch buffer size so your consumers will be
missing messages. And no, simply ignoring all IOExceptions is not the
right approach; what does it mean to ignore an EOFException or a
SocketTimeoutException?
But it seems reasonable to have a periodic retry if the networkConnector
fails. Submit an enhancement request in JIRA and maybe someone will
implement it. (Or implement it yourself and submit a patch or a pull
request, if you want it sooner.)
In the meantime, disabling statement pooling should let you work around
this. Or if you think statement pooling is an absolute must-have, you
could move to Oracle, since their driver can do its own pooling of both
connections and statements, if I'm remembering correctly.
Tim
On Mar 10, 2016 4:43 PM, "alt_alt" <al...@163.com> wrote:
> Hi Tim,
> Thanks for your useful tips. I agree that there may be some configuration
> problem in JDBS store, but I don't think the network connector should just
> be stopped and never be restarted again without restarting ActiveMQ service
> due to exeption in JDBC store. I don't think it is an acceptable behavior
> for ActiveMQ network relying on the message forwarding. Maybe you could add
> some feature to let the network still alive even such exception happens.
> For
> example, adding a peoperty of ignoreLocalIOException for networkConnector?
>
> Best Regards
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709183.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Failed network connector can not be re-established
Posted by alt_alt <al...@163.com>.
Hi Tim,
Thanks for your useful tips. I agree that there may be some configuration
problem in JDBS store, but I don't think the network connector should just
be stopped and never be restarted again without restarting ActiveMQ service
due to exeption in JDBC store. I don't think it is an acceptable behavior
for ActiveMQ network relying on the message forwarding. Maybe you could add
some feature to let the network still alive even such exception happens. For
example, adding a peoperty of ignoreLocalIOException for networkConnector?
Best Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709183.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by Tim Bain <tb...@alumni.duke.edu>.
I've never run ActiveMQ with the JDBC persistence store, so I can't answer
that question directly, and I don't get the sense that most of the people
on this mailing list have either, so you may not be able to answer that
question other than by trying it. With that being said, I wouldn't expect
it to be that large of a hit.
Tim
On Wed, Mar 9, 2016 at 5:10 PM, alt_alt <al...@163.com> wrote:
> Hi Tim,
> Really appreciated for your reply, here is the datasource configuration,
> just the same as the one shown in ActiveMQ webpage.
>
> <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
> <property name="url"
> value="jdbc:mysql://
> eanprdactivemq01writer01.eao.abn-iad.ea.com:3306/activemq?relaxAutoCommit=true
> "/>
> <property name="username" value="ACTIVEMQSERVER"/>
> <property name="password" value="${jdbc.password}"/>
> <property name="poolPreparedStatements" value="true"/>
> </bean>
>
> What's the performance impact if I set poolPreparedStatements to false?
>
> Best Regards
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709136.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Failed network connector can not be re-established
Posted by alt_alt <al...@163.com>.
Hi Tim,
Really appreciated for your reply, here is the datasource configuration,
just the same as the one shown in ActiveMQ webpage.
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url"
value="jdbc:mysql://eanprdactivemq01writer01.eao.abn-iad.ea.com:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="ACTIVEMQSERVER"/>
<property name="password" value="${jdbc.password}"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
What's the performance impact if I set poolPreparedStatements to false?
Best Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709136.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by alt_alt <al...@163.com>.
Hi Tim,
Really appreciated for your reply, here is the datasource configuration,
just the same as the one shown in ActiveMQ webpage.
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url"
value="jdbc:mysql://eanprdactivemq01writer01.eao.abn-iad.ea.com:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="ACTIVEMQSERVER"/>
<property name="password" value="${jdbc.password}"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
What's the performance impact if I set poolPreparedStatements to false?
Best Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709137.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Failed network connector can not be re-established
Posted by Tim Bain <tb...@alumni.duke.edu>.
"JDBC Failure: Borrow prepareStatement from pool failed"
Your problem is in the JDBC store (specifically the connection pool), whose
configuration you didn't post or tell us anything about.
One possible workaround would be to simply tell the DataSource to not pool
prepared statements via
https://commons.apache.org/proper/commons-dbcp/api-1.4/org/apache/commons/dbcp/BasicDataSource.html#poolPreparedStatements
.
On Mar 8, 2016 9:24 PM, "alt_alt" <al...@163.com> wrote:
> *Configuration for network connector:
> *
> <networkConnector name="external01_01" conduitSubscriptions="false"
> uri="static://(tcp://10.102.44.181:61616)" userName="MQ-PRD-ADMIN"
> password="$
> {connector.password}
> ">
> <excludedDestinations>
> <topic physicalName="VirtualTopic.>"/>
> </excludedDestinations>
> </networkConnector>
> *log:
> *2016-03-03 14:41:13,070 | WARN | JDBC Failure: Borrow prepareStatement
> from
> pool failed | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter |
> ActiveMQ Transport: tcp://10.102.44.181:61616@38805
> org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
> pool failed
> at
>
> org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRemoveMessage(DefaultJDBCAdapter.java:381)[activemq-jdbc-store-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:260)[activemq-jdbc-store-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:373)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:170)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:923)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1718)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:55)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:277)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:441)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:97)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:550)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.network.DemandForwardingBridgeSupport$11.onCompletion(DemandForwardingBridgeSupport.java:1012)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:65)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.11.1.jar:5.11.1]
> at java.lang.Thread.run(Thread.java:745)[:1.7.0_80]
> 2016-03-03 14:41:13,073 | WARN | Transport Connection to:
> vm://PRD_ACTIVEMQ_C1#0 failed: java.io.IOException: Failed to broker
> message: ID:*-36573-1456931506213-1:3:1:1:7874 in container:
> org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
> pool failed | org.apache.activemq.broker.TransportConnection.Transport |
> ActiveMQ Transport: tcp://10.102.44.181:61616@38805
> 2016-03-03 14:41:13,074 | INFO | PRD_ACTIVEMQ_C1 Shutting down |
> org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[PRD_ACTIVEMQ_C1] Task-25434
> 2016-03-03 14:41:23,080 | INFO | Network Could not shutdown in a timely
> manner | org.apache.activemq.network.DemandForwardingBridgeSupport |
> ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434
> 2016-03-03 14:41:24,083 | INFO | PRD_ACTIVEMQ_C1 bridge to localhost
> stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[PRD_ACTIVEMQ_C1] Task-25434
>
> *It is very URGENT since we found this issue on our production environment!
> Please help !!!!!!*
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>