You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Paddy O'Neill <ct...@mobilecohesion.com> on 2007/06/13 12:23:07 UTC
Performance problem possibly related to mina.
Hi.
We are having a strange problem with a production server. We have set
up mina to have 2 thread pools, one for incoming and 1 for outgoing
connections. Typically there will be multiple incoming connections
(typically around 300) and 1 outgoing connection. Traffic flows both
ways through the server.
In the production environment, we are seeing that on the outgoing
connection, a single thread is being processed for long periods of time
before another thread is worked, usually seconds but this can at times
extend to several minutes. This behaviour is causing major performance
degradation on the outgoing connection. We have built a test server
which is as close to the production server as possible, same OS, JVM
version, patches etc. and we are unable to replicate this behaviour
in-house.
We would be grateful if you could have a look at the following spring
configuration to see if there are any obvious mistakes, or failing that,
if anyone has seen similar behaviour and could point to a possible cause.
On a separate note, we are using the sessionIdle event in mina to
determine if external systems are still connected and to unbind if they
are idle for more than the configured amount of time. If we transmit
traffic to the external system, does this could as traffic for the
sessionIdle event and cause it to reset?
Paddy
Spring configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<!-- This makes it possible to specify java.net.SocketAddress values
(e.g. :80 below) as Strings.
They will be converted into java.net.InetSocketAddress objects
by Spring. -->
<bean
class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="customEditors">
<map>
<entry key="java.net.SocketAddress">
<bean
class="org.apache.mina.integration.spring.InetSocketAddressEditor"/>
</entry>
</map>
</property>
</bean>
<!-- The IoHandler delegate implementation -->
<bean id="smppHandler"
class="com.mobilecohesion.hydra.smpp.server.SMPPServerSessionHandler">
<property name="esmeSessionMgr" ref="esmeSessionMgr" />
<property name="readerIdleTimeout">
<value>600</value>
</property>
</bean>
<!-- The protocol codec factory for smpp -->
<bean id="smppProtocolCodecFactory"
class="com.mobilecohesion.hydra.smpp.server.codec.SMPPProtocolCodecFactory">
</bean>
<bean id="org.apache.mina.common.ThreadModel.MANUAL"
class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean"/>
<bean id="smppFilterThreadPoolFactory"
class="org.apache.mina.integration.spring.ThreadPoolExecutorFactoryBean">
<property name="corePoolSize" value="10"/>
<property name="maxPoolSize" value="100"/>
<property name="keepAliveSeconds" value="30"/>
</bean>
<bean id="smppFilterChainBuilder"
class="org.apache.mina.integration.spring.DefaultIoFilterChainBuilderFactoryBean">
<property name="filters">
<list>
<bean class="org.apache.mina.filter.executor.ExecutorFilter">
<constructor-arg>
<ref bean="smppFilterThreadPoolFactory"/>
</constructor-arg>
</bean>
<bean class="org.apache.mina.filter.codec.ProtocolCodecFilter">
<constructor-arg ref="smppProtocolCodecFactory"/>
</bean>
<bean class="org.apache.mina.filter.LoggingFilter"/>
</list>
</property>
<property name="filterNamePrefix" value="smppFilterPool"/>
</bean>
<bean id="smppThreadModel"
class="org.apache.mina.integration.spring.ExecutorThreadModelFactoryBean">
<property name="serviceName" value="smppService"/>
<property name="executor">
<bean
class="org.apache.mina.integration.spring.ThreadPoolExecutorFactoryBean">
<property name="corePoolSize" value="6"/>
<property name="maxPoolSize" value="10"/>
<property name="keepAliveSeconds" value="30"/>
</bean>
</property>
</bean>
<bean id="smppSocketAcceptorConfig"
class="org.apache.mina.transport.socket.nio.SocketAcceptorConfig">
<property name="filterChainBuilder" ref="smppFilterChainBuilder"/>
<property name="threadModel" ref="smppThreadModel"/>
<property name="reuseAddress" value="true"/>
</bean>
<bean id="threadExecutor"
class="java.util.concurrent.Executors"
factory-method="newCachedThreadPool"/>
<bean id="ioAcceptor"
class="org.apache.mina.integration.spring.IoAcceptorFactoryBean">
<property name="target">
<bean class="org.apache.mina.transport.socket.nio.SocketAcceptor">
<constructor-arg index="0"><value>3</value></constructor-arg>
<constructor-arg index="1"><ref
bean="threadExecutor"/></constructor-arg>
</bean>
</property>
<property name="bindings">
<list>
<bean class="org.apache.mina.integration.spring.Binding">
<property name="address" value=":2577"/>
<property name="handler" ref="smppHandler"/>
<property name="serviceConfig" ref="smppSocketAcceptorConfig"/>
</bean>
</list>
</property>
</bean>
<!-- SMSC BEANS -->
<bean id="smscFilterThreadPoolFactory"
class="org.apache.mina.integration.spring.ThreadPoolExecutorFactoryBean">
<property name="corePoolSize" value="10"/>
<property name="maxPoolSize" value="100"/>
<property name="keepAliveSeconds" value="30"/>
</bean>
<bean id="smscFilterChainBuilder"
class="org.apache.mina.integration.spring.DefaultIoFilterChainBuilderFactoryBean">
<property name="filters">
<list>
<bean class="org.apache.mina.filter.codec.ProtocolCodecFilter">
<constructor-arg ref="smppProtocolCodecFactory"/>
</bean>
<bean class="org.apache.mina.filter.executor.ExecutorFilter">
<constructor-arg>
<ref bean="smscFilterThreadPoolFactory"/>
</constructor-arg>
</bean>
<bean class="org.apache.mina.filter.LoggingFilter"/>
</list>
</property>
<property name="filterNamePrefix" value="smscFilterPool"/>
</bean>
<bean id="smscThreadModel"
class="org.apache.mina.integration.spring.ExecutorThreadModelFactoryBean">
<property name="serviceName" value="smscService"/>
<property name="executor">
<bean
class="org.apache.mina.integration.spring.ThreadPoolExecutorFactoryBean">
<property name="corePoolSize" value="6"/>
<property name="maxPoolSize" value="10"/>
<property name="keepAliveSeconds" value="30"/>
</bean>
</property>
</bean>
<bean id="smscSocketConnectorConfig"
class="org.apache.mina.transport.socket.nio.SocketConnectorConfig">
<property name="filterChainBuilder" ref="smscFilterChainBuilder"/>
<property name="threadModel" ref="smscThreadModel"/>
</bean>
</beans>
Re: Performance problem possibly related to mina.
Posted by Paddy O'Neill <ct...@mobilecohesion.com>.
Hi.
We are still having problems with this.
We have noticed that, although we are creating a number of IoProcessors
on the IoConnector side equal to number of cpu's + 1 but because we only
have a single bind on this side, there is only ever one
SocketConnectorIoProcessor thread created. If I debug the code, I can
see that the correct number of processors are being created but the
others appear never to be used. Is this normal and is there any way to
force the other IoProcessors into action?
Also, under certain circumstances we have observed that the outgoing
side, the IoConnector side, will get "stuck" on a single thread and
cause a severe drop in performance. We have tested the software using
the pre-release version of mina 1.1.1 and the thread problem does not
appear to happen. We are continuing to test this. Are there any thread
management related bug fixes in mina 1.1.1? Is this due to become
stable in the near future?
Any other help or suggestions would be appreciated.
Paddy
Paddy O'Neill wrote:
> Hi Trustin
>
> I have made the changes you suggested to the spring configuration and
> it hasn't made any obvious difference. We do not call future.join()
> but I will look at implementing the IoFutureListener that you suggested.
>
> We believe that we are now able to replicate the problem in-house. We
> have noticed that the outgoing connection only ever works on one
> thread at a time. It will switch threads every so often but never
> does anything concurrently. The inbound thread pool is actively
> working concurrently. Is this because we only have one
> IoConnectorIoProcessor thread? We create a pool of Executors equal to
> N cpus + 1 but because we only have 1 bind on the outgoing side, only
> 1 of these Executors gets used. If we put traffic through the system
> in both directions then the outgoing pool will lock onto one thread
> and constantly hammer it. This is what is causing the backlog and
> degradation in performance. Any suggestions would be gratefully
> received.
>
> Sun JVM Version 1.5.05b, Mina 1.1.0.
>
> Paddy
>
> Trustin Lee wrote:
>> On 6/13/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
>>> Hi.
>>>
>>> We are having a strange problem with a production server. We have set
>>> up mina to have 2 thread pools, one for incoming and 1 for outgoing
>>> connections. Typically there will be multiple incoming connections
>>> (typically around 300) and 1 outgoing connection. Traffic flows both
>>> ways through the server.
>>>
>>> In the production environment, we are seeing that on the outgoing
>>> connection, a single thread is being processed for long periods of time
>>> before another thread is worked, usually seconds but this can at times
>>> extend to several minutes. This behaviour is causing major performance
>>> degradation on the outgoing connection. We have built a test server
>>> which is as close to the production server as possible, same OS, JVM
>>> version, patches etc. and we are unable to replicate this behaviour
>>> in-house.
>>
>> Do you call WriteFuture.join() after writing something in your
>> IoHandler implementation? If you need to do something after the write
>> operation is complete, please try to replace join() with adding an
>> IoFutureListener to make it fully asynchronous. Any further events
>> such as messageReceived will not be processed until your current
>> handler method returns.
>>
>>> We would be grateful if you could have a look at the following spring
>>> configuration to see if there are any obvious mistakes, or failing
>>> that,
>>> if anyone has seen similar behaviour and could point to a possible
>>> cause.
>>
>> Except the following two problem, I don't see any problem with the
>> settings.
>>
>> 1) Please place protocol codec filter before the executor filter.
>> 2) Please set the thread model to MANUAL if you added an executor
>> filter with a thread pool.
>>
>>> On a separate note, we are using the sessionIdle event in mina to
>>> determine if external systems are still connected and to unbind if they
>>> are idle for more than the configured amount of time. If we transmit
>>> traffic to the external system, does this could as traffic for the
>>> sessionIdle event and cause it to reset?
>>
>> Yes. idle status is cleared when data is written out to the socket
>> channel.
>>
>> HTH,
>> Trustin
>
Re: Performance problem possibly related to mina.
Posted by Paddy O'Neill <ct...@mobilecohesion.com>.
Hi Trustin
I have made the changes you suggested to the spring configuration and it
hasn't made any obvious difference. We do not call future.join() but I
will look at implementing the IoFutureListener that you suggested.
We believe that we are now able to replicate the problem in-house. We
have noticed that the outgoing connection only ever works on one thread
at a time. It will switch threads every so often but never does
anything concurrently. The inbound thread pool is actively working
concurrently. Is this because we only have one IoConnectorIoProcessor
thread? We create a pool of Executors equal to N cpus + 1 but because
we only have 1 bind on the outgoing side, only 1 of these Executors gets
used. If we put traffic through the system in both directions then the
outgoing pool will lock onto one thread and constantly hammer it. This
is what is causing the backlog and degradation in performance. Any
suggestions would be gratefully received.
Sun JVM Version 1.5.05b, Mina 1.1.0.
Paddy
Trustin Lee wrote:
> On 6/13/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
>> Hi.
>>
>> We are having a strange problem with a production server. We have set
>> up mina to have 2 thread pools, one for incoming and 1 for outgoing
>> connections. Typically there will be multiple incoming connections
>> (typically around 300) and 1 outgoing connection. Traffic flows both
>> ways through the server.
>>
>> In the production environment, we are seeing that on the outgoing
>> connection, a single thread is being processed for long periods of time
>> before another thread is worked, usually seconds but this can at times
>> extend to several minutes. This behaviour is causing major performance
>> degradation on the outgoing connection. We have built a test server
>> which is as close to the production server as possible, same OS, JVM
>> version, patches etc. and we are unable to replicate this behaviour
>> in-house.
>
> Do you call WriteFuture.join() after writing something in your
> IoHandler implementation? If you need to do something after the write
> operation is complete, please try to replace join() with adding an
> IoFutureListener to make it fully asynchronous. Any further events
> such as messageReceived will not be processed until your current
> handler method returns.
>
>> We would be grateful if you could have a look at the following spring
>> configuration to see if there are any obvious mistakes, or failing that,
>> if anyone has seen similar behaviour and could point to a possible
>> cause.
>
> Except the following two problem, I don't see any problem with the
> settings.
>
> 1) Please place protocol codec filter before the executor filter.
> 2) Please set the thread model to MANUAL if you added an executor
> filter with a thread pool.
>
>> On a separate note, we are using the sessionIdle event in mina to
>> determine if external systems are still connected and to unbind if they
>> are idle for more than the configured amount of time. If we transmit
>> traffic to the external system, does this could as traffic for the
>> sessionIdle event and cause it to reset?
>
> Yes. idle status is cleared when data is written out to the socket
> channel.
>
> HTH,
> Trustin
Re: Performance problem possibly related to mina.
Posted by Trustin Lee <tr...@gmail.com>.
On 6/13/07, Paddy O'Neill <ct...@mobilecohesion.com> wrote:
> Hi.
>
> We are having a strange problem with a production server. We have set
> up mina to have 2 thread pools, one for incoming and 1 for outgoing
> connections. Typically there will be multiple incoming connections
> (typically around 300) and 1 outgoing connection. Traffic flows both
> ways through the server.
>
> In the production environment, we are seeing that on the outgoing
> connection, a single thread is being processed for long periods of time
> before another thread is worked, usually seconds but this can at times
> extend to several minutes. This behaviour is causing major performance
> degradation on the outgoing connection. We have built a test server
> which is as close to the production server as possible, same OS, JVM
> version, patches etc. and we are unable to replicate this behaviour
> in-house.
Do you call WriteFuture.join() after writing something in your
IoHandler implementation? If you need to do something after the write
operation is complete, please try to replace join() with adding an
IoFutureListener to make it fully asynchronous. Any further events
such as messageReceived will not be processed until your current
handler method returns.
> We would be grateful if you could have a look at the following spring
> configuration to see if there are any obvious mistakes, or failing that,
> if anyone has seen similar behaviour and could point to a possible cause.
Except the following two problem, I don't see any problem with the settings.
1) Please place protocol codec filter before the executor filter.
2) Please set the thread model to MANUAL if you added an executor
filter with a thread pool.
> On a separate note, we are using the sessionIdle event in mina to
> determine if external systems are still connected and to unbind if they
> are idle for more than the configured amount of time. If we transmit
> traffic to the external system, does this could as traffic for the
> sessionIdle event and cause it to reset?
Yes. idle status is cleared when data is written out to the socket channel.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6