You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Paul French <pa...@frenchiesystems.com> on 2007/02/08 17:02:29 UTC
Async Exception, multiple threads, one connection
For me this is a continuation of.....
http://www.nabble.com/Async-Exception%3A-Broken-Pipe-t3160498s2354.html#a8765808
...but I have now taking the advice from the above post and have refined the
problem I am having.
Environment
I am now using the latest 4.2 snapshot as requested (29th Jan 2007)
I am also running the broker on its own dedicated machine.
I am also using Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
on all machines.
Configuration
ActiveMQ broker is configured with no persistence and no transactions are
used. Async messaging is used.
Problem
I have a program that simply starts as many threads (non-daemon) as you
require (I use 20). Once all threads have been started the main thread
simply finishes. Each thread does the following:
Gets a connection
connection = singleConnectionFactory.createConnection();
where the following spring context snippet is used
<bean id="connectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
destroy-method="destroy">
<property name="targetConnectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="tcp://192.168.160.86:61616?jms.prefetchPolicy.queuePrefetch=1" />
<property name="useAsyncSend"><value>TRUE</value></property>
</bean>
</property>
</bean>
Please note that each thread is given the same connection since I am using a
SingleConnectionFactory. Each thread then runs in a continuous loop doing
nothing of interest. I do not create any sessions, producers or consumers
and I do not send any messages. I do not touch the connection whatsoever!!!
then after five minutes or so my program reports the Exception..........
WARN [08/02/2007 11:47:01]: Async exception with no exception listener:
java.net.SocketException: Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
at
org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:81)
at
org.apache.activemq.transport.InactivityMonitor.access$100(InactivityMonitor.java:35)
at
org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:57)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
The wierd thing is I know my program is about to report this Exception since
all threads in my program stop doing any activity for about 25 seconds.
During this 25 seconds the above Exception is thrown. Then all my threads
carry on as normal.
I do not understand why having a single connection open should cause such
problems and stop all my threads. Any ideas? I know this problem does not
seem to relate to a real life situation but I have factored it out as a
problem from a real situation and tried to simplify the example.
--
View this message in context: http://www.nabble.com/Async-Exception%2C-multiple-threads%2C-one-connection-tf3194160s2354.html#a8868199
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Async Exception, multiple threads, one connection
Posted by "Christopher G. Stach II" <cg...@ldsys.net>.
Paul French wrote:
>
> Okay, I switched on garbage collection output using the flag -verbosegc and
> the Aysnc failure on the client does coincide with a FULL GC event. On my
> client system I am close to the max heap size nearly all the time. The FULL
> GC takes about 25 seconds to complete. All client threads are suspended
> during this time. I assume then that the broker is not recieving heartbeat
> messages of some sort from the client program and so closes the connection.
> Again it appears from the broker log this is the case. Hence when the client
> starts again the connection has been lost. Hence the ultimate solution is to
> stop the FULL GC happening in the first place!!
>
> However if I set the connection url to use failover can someone explain to
> me what happens if i have created 20 sessions (where each session has a one
> consumer and one producer) against this connection and then the connection
> fails but is reconnected?
>
> Does my client program need to trap any Exceptions to continue or will all
> the sessions, consumers and producers still be usable even after the
> connection has been re-established?
>
> What happens to any temporary queues I have created. Temporary queues are
> unique per connection. Hence when a connection is re-established will
> ActiveMQ maintain my current temporary queues?
>
Use the CMS collector and parallel collectors for the other generations.
--
Christopher G. Stach II
Re: Async Exception, multiple threads, one connection
Posted by James Strachan <ja...@gmail.com>.
So yes - its looking like the connection is being disconnected due to
inactivity. You could increase the inactivity timeout to something
much bigger (or even disable it) to avoid the broker closing your
connections.
Whether the connection is closed or not, with failover: all
sessions/producers/consumers/destinations are recreated for you
On 2/8/07, Paul French <pa...@frenchiesystems.com> wrote:
>
>
> Okay, I switched on garbage collection output using the flag -verbosegc and
> the Aysnc failure on the client does coincide with a FULL GC event. On my
> client system I am close to the max heap size nearly all the time. The FULL
> GC takes about 25 seconds to complete. All client threads are suspended
> during this time. I assume then that the broker is not recieving heartbeat
> messages of some sort from the client program and so closes the connection.
> Again it appears from the broker log this is the case. Hence when the client
> starts again the connection has been lost. Hence the ultimate solution is to
> stop the FULL GC happening in the first place!!
>
> However if I set the connection url to use failover can someone explain to
> me what happens if i have created 20 sessions (where each session has a one
> consumer and one producer) against this connection and then the connection
> fails but is reconnected?
>
> Does my client program need to trap any Exceptions to continue or will all
> the sessions, consumers and producers still be usable even after the
> connection has been re-established?
>
> What happens to any temporary queues I have created. Temporary queues are
> unique per connection. Hence when a connection is re-established will
> ActiveMQ maintain my current temporary queues?
>
>
>
>
> Paul French wrote:
> >
> > For me this is a continuation of.....
> >
> > http://www.nabble.com/Async-Exception%3A-Broken-Pipe-t3160498s2354.html#a8765808
> >
> > ...but I have now taking the advice from the above post and have refined
> > the problem I am having.
> >
> >
> > Environment
> > I am now using the latest 4.2 snapshot as requested (29th Jan 2007)
> > I am also running the broker on its own dedicated machine.
> > I am also using Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed
> > mode) on all machines.
> >
> > Configuration
> > ActiveMQ broker is configured with no persistence and no transactions are
> > used. Async messaging is used.
> >
> > Problem
> > I have a program that simply starts as many threads (non-daemon) as you
> > require (I use 20). Once all threads have been started the main thread
> > simply finishes. Each thread does the following:
> >
> > Gets a connection
> >
> > connection = singleConnectionFactory.createConnection();
> >
> > where the following spring context snippet is used
> >
> > <bean id="connectionFactory"
> > class="org.springframework.jms.connection.SingleConnectionFactory"
> > destroy-method="destroy">
> > <property name="targetConnectionFactory">
> > <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> > <property name="brokerURL"
> > value="tcp://192.168.160.86:61616?jms.prefetchPolicy.queuePrefetch=1" />
> > <property name="useAsyncSend"><value>TRUE</value></property>
> > </bean>
> > </property>
> > </bean>
> >
> > Please note that each thread is given the same connection since I am using
> > a SingleConnectionFactory. Each thread then runs in a continuous loop
> > doing nothing of interest. I do not create any sessions, producers or
> > consumers and I do not send any messages. I do not touch the connection
> > whatsoever!!!
> >
> > then after five minutes or so my program reports the Exception..........
> >
> > WARN [08/02/2007 11:47:01]: Async exception with no exception listener:
> > java.net.SocketException: Broken pipe
> > java.net.SocketException: Broken pipe
> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> > at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> > at
> > org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
> > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > at
> > org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
> > at
> > org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:81)
> > at
> > org.apache.activemq.transport.InactivityMonitor.access$100(InactivityMonitor.java:35)
> > at
> > org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:57)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> > at java.lang.Thread.run(Thread.java:595)
> >
> > The wierd thing is I know my program is about to report this Exception
> > since all threads in my program stop doing any activity for about 25
> > seconds. During this 25 seconds the above Exception is thrown. Then all my
> > threads carry on as normal.
> >
> > I do not understand why having a single connection open should cause such
> > problems and stop all my threads. Any ideas? I know this problem does not
> > seem to relate to a real life situation but I have factored it out as a
> > problem from a real situation and tried to simplify the example.
> >
> >
> >
> >
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Async-Exception%2C-multiple-threads%2C-one-connection-tf3194160s2354.html#a8870982
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Async Exception, multiple threads, one connection
Posted by Paul French <pa...@frenchiesystems.com>.
Okay, I switched on garbage collection output using the flag -verbosegc and
the Aysnc failure on the client does coincide with a FULL GC event. On my
client system I am close to the max heap size nearly all the time. The FULL
GC takes about 25 seconds to complete. All client threads are suspended
during this time. I assume then that the broker is not recieving heartbeat
messages of some sort from the client program and so closes the connection.
Again it appears from the broker log this is the case. Hence when the client
starts again the connection has been lost. Hence the ultimate solution is to
stop the FULL GC happening in the first place!!
However if I set the connection url to use failover can someone explain to
me what happens if i have created 20 sessions (where each session has a one
consumer and one producer) against this connection and then the connection
fails but is reconnected?
Does my client program need to trap any Exceptions to continue or will all
the sessions, consumers and producers still be usable even after the
connection has been re-established?
What happens to any temporary queues I have created. Temporary queues are
unique per connection. Hence when a connection is re-established will
ActiveMQ maintain my current temporary queues?
Paul French wrote:
>
> For me this is a continuation of.....
>
> http://www.nabble.com/Async-Exception%3A-Broken-Pipe-t3160498s2354.html#a8765808
>
> ...but I have now taking the advice from the above post and have refined
> the problem I am having.
>
>
> Environment
> I am now using the latest 4.2 snapshot as requested (29th Jan 2007)
> I am also running the broker on its own dedicated machine.
> I am also using Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed
> mode) on all machines.
>
> Configuration
> ActiveMQ broker is configured with no persistence and no transactions are
> used. Async messaging is used.
>
> Problem
> I have a program that simply starts as many threads (non-daemon) as you
> require (I use 20). Once all threads have been started the main thread
> simply finishes. Each thread does the following:
>
> Gets a connection
>
> connection = singleConnectionFactory.createConnection();
>
> where the following spring context snippet is used
>
> <bean id="connectionFactory"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> destroy-method="destroy">
> <property name="targetConnectionFactory">
> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL"
> value="tcp://192.168.160.86:61616?jms.prefetchPolicy.queuePrefetch=1" />
> <property name="useAsyncSend"><value>TRUE</value></property>
> </bean>
> </property>
> </bean>
>
> Please note that each thread is given the same connection since I am using
> a SingleConnectionFactory. Each thread then runs in a continuous loop
> doing nothing of interest. I do not create any sessions, producers or
> consumers and I do not send any messages. I do not touch the connection
> whatsoever!!!
>
> then after five minutes or so my program reports the Exception..........
>
> WARN [08/02/2007 11:47:01]: Async exception with no exception listener:
> java.net.SocketException: Broken pipe
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
> at
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:81)
> at
> org.apache.activemq.transport.InactivityMonitor.access$100(InactivityMonitor.java:35)
> at
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:57)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
>
> The wierd thing is I know my program is about to report this Exception
> since all threads in my program stop doing any activity for about 25
> seconds. During this 25 seconds the above Exception is thrown. Then all my
> threads carry on as normal.
>
> I do not understand why having a single connection open should cause such
> problems and stop all my threads. Any ideas? I know this problem does not
> seem to relate to a real life situation but I have factored it out as a
> problem from a real situation and tried to simplify the example.
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Async-Exception%2C-multiple-threads%2C-one-connection-tf3194160s2354.html#a8870982
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Async Exception, multiple threads, one connection
Posted by Paul French <pa...@frenchiesystems.com>.
Hold fire on this one, it could be a garbage collection problem causing the
long pause. I've got parallel garbage collection switched on so I would not
expect all threads to stop so it is a strange one.................
Paul French wrote:
>
> For me this is a continuation of.....
>
> http://www.nabble.com/Async-Exception%3A-Broken-Pipe-t3160498s2354.html#a8765808
>
> ...but I have now taking the advice from the above post and have refined
> the problem I am having.
>
>
> Environment
> I am now using the latest 4.2 snapshot as requested (29th Jan 2007)
> I am also running the broker on its own dedicated machine.
> I am also using Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed
> mode) on all machines.
>
> Configuration
> ActiveMQ broker is configured with no persistence and no transactions are
> used. Async messaging is used.
>
> Problem
> I have a program that simply starts as many threads (non-daemon) as you
> require (I use 20). Once all threads have been started the main thread
> simply finishes. Each thread does the following:
>
> Gets a connection
>
> connection = singleConnectionFactory.createConnection();
>
> where the following spring context snippet is used
>
> <bean id="connectionFactory"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> destroy-method="destroy">
> <property name="targetConnectionFactory">
> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL"
> value="tcp://192.168.160.86:61616?jms.prefetchPolicy.queuePrefetch=1" />
> <property name="useAsyncSend"><value>TRUE</value></property>
> </bean>
> </property>
> </bean>
>
> Please note that each thread is given the same connection since I am using
> a SingleConnectionFactory. Each thread then runs in a continuous loop
> doing nothing of interest. I do not create any sessions, producers or
> consumers and I do not send any messages. I do not touch the connection
> whatsoever!!!
>
> then after five minutes or so my program reports the Exception..........
>
> WARN [08/02/2007 11:47:01]: Async exception with no exception listener:
> java.net.SocketException: Broken pipe
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
> at
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:81)
> at
> org.apache.activemq.transport.InactivityMonitor.access$100(InactivityMonitor.java:35)
> at
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:57)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
>
> The wierd thing is I know my program is about to report this Exception
> since all threads in my program stop doing any activity for about 25
> seconds. During this 25 seconds the above Exception is thrown. Then all my
> threads carry on as normal.
>
> I do not understand why having a single connection open should cause such
> problems and stop all my threads. Any ideas? I know this problem does not
> seem to relate to a real life situation but I have factored it out as a
> problem from a real situation and tried to simplify the example.
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Async-Exception%2C-multiple-threads%2C-one-connection-tf3194160s2354.html#a8869559
Sent from the ActiveMQ - User mailing list archive at Nabble.com.