You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by jhcha <jh...@kds.co.kr> on 2008/02/04 11:51:39 UTC
sessionClosed is not called sometimes in heavy stress test
contions.
Dear Mina developers.
I'm trying to use mina 1.1.5 to develop the very big stress server.
I test my server with jmeter like some echo test now.
test environment
* jmeter :
echo message size : 1024 bytes
jmeter threads : 100
loop count : 1000
sampler : TCP Sampler
reuse connection : false
* server : HP-UX 2 cores
java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
IA64, mixed mode)
my server(mina)
* client : pc (dual core).
jmeter
when I tested the stress, I found sessionCreated called counts are a little
bigger than sessionClosed called counts.
so, I tried to find out why that mismatch occured during several days.
and I found the followings :
org.apache.mina.common.support.IoServiceListenerSupport
{
public void fireSessionCreated(IoSession session) {
SocketAddress serviceAddress = session.getServiceAddress();
// Get the session set.
Set<IoSession> s = new IdentityHashSet<IoSession>();
Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
Collections
.synchronizedSet(s));
boolean firstSession;
if (null == sessions) {
sessions = s;
firstSession = true;
} else {
firstSession = false;
}
// If already registered, ignore.
if (!sessions.add(session)) {
return;
}
// If the first connector session, fire a virtual service activation
// event.
if (session.getService() instanceof IoConnector && firstSession) {
fireServiceActivated(session.getService(), session.getServiceAddress(),
session
.getHandler(), session.getServiceConfig());
}
// Fire session events.
session.getFilterChain().fireSessionCreated(session); <-- called
correctly
session.getFilterChain().fireSessionOpened(session); <-- called correctly
// Fire listener events.
for (IoServiceListener listener : listeners) {
listener.sessionCreated(session); <-- called correctly
}
}
public void fireSessionDestroyed(IoSession session) {
boolean lastSession = false;
SocketAddress serviceAddress = session.getServiceAddress();
// Get the session set.
Set<IoSession> sessions = managedSessions.get(serviceAddress);
// Ignore if unknown.
if (sessions == null) {
// logger.error("fireSessionDestroyed() : sessions == null < " +
session);
return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
}
sessions.remove(session);
// Try to remove the remaining empty session set after removal.
if (sessions.isEmpty()) {
lastSession = managedSessions.remove(serviceAddress, sessions);
}
// Fire session events.
session.getFilterChain().fireSessionClosed(session); <-- skipped sometimes
// Fire listener events.
try {
for (IoServiceListener listener : listeners) {
listener.sessionDestroyed(session); <-- skipped sometimes
}
} finally {
// Fire a virtual service deactivation event for the last session of
// the connector.
// TODO double-check that this is *STILL* the last session. May not
// be the case
if (session.getService() instanceof IoConnector && lastSession) {
fireServiceDeactivated(session.getService(),
session.getServiceAddress(), session
.getHandler(), session.getServiceConfig()); <-- skipped sometimes
}
}
}
the above fireSessionDestroyed method called as the same number as
fireSessionCreated method.
but the mysterious event is that the sessions is not gotten from
managedSessions in fireSessionDestroyed,
i.e. SocketAddress key is not found in managedSessions.
then the remaining logic was skipped.
so fireSessionClosed, sessionDestroyed ... are not called sometimes.
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
[IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
(SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
0.0.0.0/0.0.0.0:30000)
I think the reason that the 100 client threads tries to proceed "conect,
send, receive, close",
different sessions has the same client SocketAddress at times in server,
and the origin of that problem is client's stress environment.
It is possible the following senario :
I/O processor thread I/O processor thread client thread client
thread
one two one two
--------------------------------------------------------------------------------
1) connect(the same
client SocketAddress)
2)fireSessionCreated
3)putIfAbsent
4)disconnect
5)connect(the same client SocketAddress)
6)fireSessionCreated
7)putIfAbsent
8)fireSessionDestroyed
9)remove
10)disconnect
11)fireSessionDestroyed
12)get fail
I changed the mina source IoServiceListenerSupport.java
org.apache.mina.common.support.IoServiceListenerSupport
...
public void fireSessionDestroyed(IoSession session) {
boolean lastSession = false;
SocketAddress serviceAddress = session.getServiceAddress();
// Get the session set.
Set<IoSession> sessions = managedSessions.get(serviceAddress);
// Ignore if unknown.
if (sessions == null) {
// return; <-- delete
} else { // <-- add
sessions.remove(session);
// Try to remove the remaining empty session set after removal.
if (sessions.isEmpty()) {
lastSession = managedSessions.remove(serviceAddress, sessions);
}
} // <-- add
....
then the Create, Destroy call mismatch was cleared.
but I don't know this is correct solution.
Would you tell me this is correct or not?
I think this situation is very rare but...
I hope the mina is the best network framework
and this situation shoud be also considered.
Thank you
J. H. Cha
--
View this message in context: http://www.nabble.com/sessionClosed-is-not-called-sometimes-in-heavy-stress-test-contions.-tp15265671s16868p15265671.html
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: sessionClosed is not called sometimes in heavy stress test
contions.
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
Your analysis is correct, and I've just checked in the fix. However, I
don't think your issue is related with MINA 2, because it uses a
different way to manage a session set. Please check if my fix fixes
your problem and close the following issue:
https://issues.apache.org/jira/browse/DIRMINA-549
Thanks,
2008-02-07 (목), 23:59 -0800, jhcha 쓰시길:
> Thank Mike for replying my question.
>
> I read your answers but...
>
> I think MINA already knew the session was being closed in my questioned
> situation.
> because my server session always close after sending the echo data and
> I debugged MINA source
> org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method
> always call doRemove() at (SocketIoProcessor.java:489 line).
> then doRemove method always call
> session.getServiceListeners().fireSessionDestroyed(session);
> at (SocketIoProcessor.java:188 line).
>
> try {
> key.cancel();
> ch.close();
> } catch (IOException e) {
> session.getFilterChain().fireExceptionCaught(session, e);
> } finally {
> releaseWriteBuffers(session);
> session.getServiceListeners().fireSessionDestroyed(session); <---
> SocketIoProcessor.java:188 line
> }
>
>
> and the above session parameter of the fireSessionDestroyed method was
> always not null.
>
> the orgin of the problem is the sessions having serviceAddress is not in
> managedSessions hashmap
> in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) {
> return;
> }
>
> The fireSessionDestroyed method call propagation stopped only sometimes
> Because sessions is not gotten from managedSessions at
> (IoServiceListenerSupport.java:192 line).
>
> I used setIdleTimeout with read mode, but
> if sessionClosed method is not called, sessionIdle or sessionClosed method
> will never be called after that.
>
> My question is that the filter chain fireSessionClosed method and listener'
> sessionDestroyed method
> Should not know if the destroying session be not in the managedSessions
> hashmap ?
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session); <---
> IoServiceListenerSupport.java:208 line
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session); <---
> }
> }
>
>
> If fireSessionDestroyed method were called with the parameter the closing
> session
> but not in managedSessions hashmap, Should filter chain fireSessionClosed
> method and
> listener' sessionDestroyed method not called ?
>
>
> in the following MINA source below the IoServiceListenerSupport.java:147
> line.
>
>
> public void fireSessionCreated(IoSession session) {
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> s = new IdentityHashSet<IoSession>();
> Set<IoSession> sessions =
> managedSessions.putIfAbsent(serviceAddress,
> Collections.synchronizedSet(s));
> boolean firstSession;
>
> if (null == sessions) {
> sessions = s;
> firstSession = true;
> } else {
> firstSession = false;
> }
>
> // If already registered, ignore.
> if (!sessions.add(session)) { <---
> return;
> }
>
> // If the first connector session, fire a virtual service activation
> event.
> if (session.getService() instanceof IoConnector && firstSession) {
> fireServiceActivated(session.getService(), session
> .getServiceAddress(), session.getHandler(), session
> .getServiceConfig());
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionCreated(session);
> session.getFilterChain().fireSessionOpened(session);
>
> // Fire listener events.
> for (IoServiceListener listener : listeners) {
> listener.sessionCreated(session);
> }
> }
>
>
> I think the code "if (!sessions.add(session))" is always true,
> So filter chain fireSessionCreated method and listener' sessionCreated
> method is
> always called in spite of the heavy unstable tcp stress condition.
>
> I think
> Creaing callback method and destroying callback method should have the same
> calling count.
>
> But creating callback logic and managedSessions hashmap are not related
> directly
> in the above fireSessionCreated method.
>
> "not related directly" means that
> whether sessions put managedSessions hashmap or not, fireSessionCreated and
> sessionCreated are called.
>
> Then destroying callback logic and managedSessions hashmap should be also
> not related directly.
> but MINA has the direct relation with managedSessions hashmap
>
> "direct relation" means that
> as the below code shows, if sessions is not found in managedSessions hashmap
> then
> fireSessionClosed and sessionDestroyed method are not called.
>
>
> in the code "public void fireSessionDestroyed(IoSession session)" method
> of IoServiceListenerSupport.java
>
> ...
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) { <---
> return;
> }
>
> sessions.remove(session);
>
> boolean lastSession = false;
>
> // Try to remove the remaining empty session set after removal.
> if (sessions.isEmpty()) {
> lastSession = managedSessions.remove(serviceAddress, sessions);
> <====
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session);
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session);
> }
> }
>
> ...
>
> My inference is that
> When heavy thread competition (and other session parameters but with the
> same serviceAddress),
> the one thread can remove the other using sessions from managedSessions
> hashmap.
>
> 1) the one thread : sessions.isEmpty() succeeded in fireSessionDestroyed
> 2) the other thread : called fireSessionCreated
> 3) the other thread : sessions.add(session) suceeded in fireSessionCreated
> 4) the one thread : managedSessions.remove(serviceAddress, sessions);
> (above "<===" line)
> 5) the other thread : called fireSessionDestroyed
> 6) the other thread : managedSessions.get(serviceAddress); fail
> (above "<---" line)
>
> other thread fails to get sessions from managedSessions hashmap with the
> serviceAddress .
> then fireSessionClosed propagation stops. (but should it continue!)
>
>
> It is just my inference.
> and it's possible for me to misunderstand the MINA logic.
>
>
> Thank you again.
>
>
> J. H. Cha
>
>
>
> Mike Heath-4 wrote:
> >
> > It's possible that the server never received the TCP FIN packet and
> > therefor did not know that it should close the IoSession. There are
> > situations where a host has no way of knowing that the TCP socket has
> > closed unless it tries to write data to the socket and that write fails.
> > This is not a limitation of MINA. This is just the way socket I/O works.
> >
> > If this is the cause of your problem, you have a couple of options. You
> > could use some type of watch dog mechanisms that sends data at a regular
> > interval. If the connection is no longer valid, the write will fail and
> > the socket will get closed thereby triggering a sessionClosed event. Or
> > you could use the MINA read idle notification. This way after a certain
> > amount of time has passed and your server hasn't received any data for a
> > particular IoSession, you can manually close the IoSession.
> >
> > If your problem still persists after implementing one of these
> > solutions, please let us know.
> >
> > -Mike
> >
> > jhcha wrote:
> >> Dear Mina developers.
> >> I'm trying to use mina 1.1.5 to develop the very big stress server.
> >> I test my server with jmeter like some echo test now.
> >>
> >> test environment
> >>
> >> * jmeter :
> >> echo message size : 1024 bytes
> >> jmeter threads : 100
> >> loop count : 1000
> >> sampler : TCP Sampler
> >> reuse connection : false
> >>
> >> * server : HP-UX 2 cores
> >> java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> >> IA64, mixed mode)
> >> my server(mina)
> >>
> >> * client : pc (dual core).
> >> jmeter
> >>
> >> when I tested the stress, I found sessionCreated called counts are a
> >> little
> >> bigger than sessionClosed called counts.
> >> so, I tried to find out why that mismatch occured during several days.
> >> and I found the followings :
> >>
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> {
> >>
> >> public void fireSessionCreated(IoSession session) {
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> s = new IdentityHashSet<IoSession>();
> >> Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
> >> Collections
> >> .synchronizedSet(s));
> >> boolean firstSession;
> >>
> >> if (null == sessions) {
> >> sessions = s;
> >> firstSession = true;
> >> } else {
> >> firstSession = false;
> >> }
> >>
> >> // If already registered, ignore.
> >> if (!sessions.add(session)) {
> >> return;
> >> }
> >>
> >> // If the first connector session, fire a virtual service activation
> >> // event.
> >> if (session.getService() instanceof IoConnector && firstSession) {
> >> fireServiceActivated(session.getService(),
> >> session.getServiceAddress(),
> >> session
> >> .getHandler(), session.getServiceConfig());
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionCreated(session); <-- called
> >> correctly
> >> session.getFilterChain().fireSessionOpened(session); <-- called
> >> correctly
> >>
> >> // Fire listener events.
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionCreated(session); <-- called correctly
> >> }
> >> }
> >>
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // logger.error("fireSessionDestroyed() : sessions == null < " +
> >> session);
> >> return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
> >> }
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after removal.
> >> if (sessions.isEmpty()) {
> >> lastSession = managedSessions.remove(serviceAddress, sessions);
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionClosed(session); <-- skipped
> >> sometimes
> >>
> >> // Fire listener events.
> >> try {
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionDestroyed(session); <-- skipped sometimes
> >> }
> >> } finally {
> >> // Fire a virtual service deactivation event for the last session of
> >> // the connector.
> >> // TODO double-check that this is *STILL* the last session. May not
> >> // be the case
> >> if (session.getService() instanceof IoConnector && lastSession) {
> >> fireServiceDeactivated(session.getService(),
> >> session.getServiceAddress(), session
> >> .getHandler(), session.getServiceConfig()); <-- skipped sometimes
> >> }
> >> }
> >> }
> >>
> >> the above fireSessionDestroyed method called as the same number as
> >> fireSessionCreated method.
> >>
> >> but the mysterious event is that the sessions is not gotten from
> >> managedSessions in fireSessionDestroyed,
> >> i.e. SocketAddress key is not found in managedSessions.
> >> then the remaining logic was skipped.
> >> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
> >>
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >>
> >> I think the reason that the 100 client threads tries to proceed "conect,
> >> send, receive, close",
> >> different sessions has the same client SocketAddress at times in server,
> >> and the origin of that problem is client's stress environment.
> >>
> >> It is possible the following senario :
> >>
> >> I/O processor thread I/O processor thread client thread client
> >> thread
> >> one two one two
> >> --------------------------------------------------------------------------------
> >> 1) connect(the same
> >> client SocketAddress)
> >> 2)fireSessionCreated
> >> 3)putIfAbsent
> >> 4)disconnect
> >>
> >> 5)connect(the same client SocketAddress)
> >> 6)fireSessionCreated
> >> 7)putIfAbsent
> >> 8)fireSessionDestroyed
> >> 9)remove
> >>
> >> 10)disconnect
> >> 11)fireSessionDestroyed
> >> 12)get fail
> >>
> >>
> >>
> >> I changed the mina source IoServiceListenerSupport.java
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> ...
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // return; <-- delete
> >> } else { // <-- add
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after removal.
> >> if (sessions.isEmpty()) {
> >> lastSession = managedSessions.remove(serviceAddress, sessions);
> >> }
> >> } // <-- add
> >> ....
> >>
> >> then the Create, Destroy call mismatch was cleared.
> >> but I don't know this is correct solution.
> >>
> >> Would you tell me this is correct or not?
> >>
> >> I think this situation is very rare but...
> >> I hope the mina is the best network framework
> >> and this situation shoud be also considered.
> >>
> >>
> >> Thank you
> >>
> >> J. H. Cha
> >
> >
> >
>
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: sessionClosed is not called sometimes in heavy stress test
contions.
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
JH,
Thank you so much for the detailed analysis. Let me take a look into
this issue when I get back home.
2008-02-07 (목), 23:59 -0800, jhcha 쓰시길:
> Thank Mike for replying my question.
>
> I read your answers but...
>
> I think MINA already knew the session was being closed in my questioned
> situation.
> because my server session always close after sending the echo data and
> I debugged MINA source
> org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method
> always call doRemove() at (SocketIoProcessor.java:489 line).
> then doRemove method always call
> session.getServiceListeners().fireSessionDestroyed(session);
> at (SocketIoProcessor.java:188 line).
>
> try {
> key.cancel();
> ch.close();
> } catch (IOException e) {
> session.getFilterChain().fireExceptionCaught(session, e);
> } finally {
> releaseWriteBuffers(session);
> session.getServiceListeners().fireSessionDestroyed(session); <---
> SocketIoProcessor.java:188 line
> }
>
>
> and the above session parameter of the fireSessionDestroyed method was
> always not null.
>
> the orgin of the problem is the sessions having serviceAddress is not in
> managedSessions hashmap
> in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) {
> return;
> }
>
> The fireSessionDestroyed method call propagation stopped only sometimes
> Because sessions is not gotten from managedSessions at
> (IoServiceListenerSupport.java:192 line).
>
> I used setIdleTimeout with read mode, but
> if sessionClosed method is not called, sessionIdle or sessionClosed method
> will never be called after that.
>
> My question is that the filter chain fireSessionClosed method and listener'
> sessionDestroyed method
> Should not know if the destroying session be not in the managedSessions
> hashmap ?
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session); <---
> IoServiceListenerSupport.java:208 line
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session); <---
> }
> }
>
>
> If fireSessionDestroyed method were called with the parameter the closing
> session
> but not in managedSessions hashmap, Should filter chain fireSessionClosed
> method and
> listener' sessionDestroyed method not called ?
>
>
> in the following MINA source below the IoServiceListenerSupport.java:147
> line.
>
>
> public void fireSessionCreated(IoSession session) {
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> s = new IdentityHashSet<IoSession>();
> Set<IoSession> sessions =
> managedSessions.putIfAbsent(serviceAddress,
> Collections.synchronizedSet(s));
> boolean firstSession;
>
> if (null == sessions) {
> sessions = s;
> firstSession = true;
> } else {
> firstSession = false;
> }
>
> // If already registered, ignore.
> if (!sessions.add(session)) { <---
> return;
> }
>
> // If the first connector session, fire a virtual service activation
> event.
> if (session.getService() instanceof IoConnector && firstSession) {
> fireServiceActivated(session.getService(), session
> .getServiceAddress(), session.getHandler(), session
> .getServiceConfig());
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionCreated(session);
> session.getFilterChain().fireSessionOpened(session);
>
> // Fire listener events.
> for (IoServiceListener listener : listeners) {
> listener.sessionCreated(session);
> }
> }
>
>
> I think the code "if (!sessions.add(session))" is always true,
> So filter chain fireSessionCreated method and listener' sessionCreated
> method is
> always called in spite of the heavy unstable tcp stress condition.
>
> I think
> Creaing callback method and destroying callback method should have the same
> calling count.
>
> But creating callback logic and managedSessions hashmap are not related
> directly
> in the above fireSessionCreated method.
>
> "not related directly" means that
> whether sessions put managedSessions hashmap or not, fireSessionCreated and
> sessionCreated are called.
>
> Then destroying callback logic and managedSessions hashmap should be also
> not related directly.
> but MINA has the direct relation with managedSessions hashmap
>
> "direct relation" means that
> as the below code shows, if sessions is not found in managedSessions hashmap
> then
> fireSessionClosed and sessionDestroyed method are not called.
>
>
> in the code "public void fireSessionDestroyed(IoSession session)" method
> of IoServiceListenerSupport.java
>
> ...
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) { <---
> return;
> }
>
> sessions.remove(session);
>
> boolean lastSession = false;
>
> // Try to remove the remaining empty session set after removal.
> if (sessions.isEmpty()) {
> lastSession = managedSessions.remove(serviceAddress, sessions);
> <====
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session);
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session);
> }
> }
>
> ...
>
> My inference is that
> When heavy thread competition (and other session parameters but with the
> same serviceAddress),
> the one thread can remove the other using sessions from managedSessions
> hashmap.
>
> 1) the one thread : sessions.isEmpty() succeeded in fireSessionDestroyed
> 2) the other thread : called fireSessionCreated
> 3) the other thread : sessions.add(session) suceeded in fireSessionCreated
> 4) the one thread : managedSessions.remove(serviceAddress, sessions);
> (above "<===" line)
> 5) the other thread : called fireSessionDestroyed
> 6) the other thread : managedSessions.get(serviceAddress); fail
> (above "<---" line)
>
> other thread fails to get sessions from managedSessions hashmap with the
> serviceAddress .
> then fireSessionClosed propagation stops. (but should it continue!)
>
>
> It is just my inference.
> and it's possible for me to misunderstand the MINA logic.
>
>
> Thank you again.
>
>
> J. H. Cha
>
>
>
> Mike Heath-4 wrote:
> >
> > It's possible that the server never received the TCP FIN packet and
> > therefor did not know that it should close the IoSession. There are
> > situations where a host has no way of knowing that the TCP socket has
> > closed unless it tries to write data to the socket and that write fails.
> > This is not a limitation of MINA. This is just the way socket I/O works.
> >
> > If this is the cause of your problem, you have a couple of options. You
> > could use some type of watch dog mechanisms that sends data at a regular
> > interval. If the connection is no longer valid, the write will fail and
> > the socket will get closed thereby triggering a sessionClosed event. Or
> > you could use the MINA read idle notification. This way after a certain
> > amount of time has passed and your server hasn't received any data for a
> > particular IoSession, you can manually close the IoSession.
> >
> > If your problem still persists after implementing one of these
> > solutions, please let us know.
> >
> > -Mike
> >
> > jhcha wrote:
> >> Dear Mina developers.
> >> I'm trying to use mina 1.1.5 to develop the very big stress server.
> >> I test my server with jmeter like some echo test now.
> >>
> >> test environment
> >>
> >> * jmeter :
> >> echo message size : 1024 bytes
> >> jmeter threads : 100
> >> loop count : 1000
> >> sampler : TCP Sampler
> >> reuse connection : false
> >>
> >> * server : HP-UX 2 cores
> >> java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> >> IA64, mixed mode)
> >> my server(mina)
> >>
> >> * client : pc (dual core).
> >> jmeter
> >>
> >> when I tested the stress, I found sessionCreated called counts are a
> >> little
> >> bigger than sessionClosed called counts.
> >> so, I tried to find out why that mismatch occured during several days.
> >> and I found the followings :
> >>
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> {
> >>
> >> public void fireSessionCreated(IoSession session) {
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> s = new IdentityHashSet<IoSession>();
> >> Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
> >> Collections
> >> .synchronizedSet(s));
> >> boolean firstSession;
> >>
> >> if (null == sessions) {
> >> sessions = s;
> >> firstSession = true;
> >> } else {
> >> firstSession = false;
> >> }
> >>
> >> // If already registered, ignore.
> >> if (!sessions.add(session)) {
> >> return;
> >> }
> >>
> >> // If the first connector session, fire a virtual service activation
> >> // event.
> >> if (session.getService() instanceof IoConnector && firstSession) {
> >> fireServiceActivated(session.getService(),
> >> session.getServiceAddress(),
> >> session
> >> .getHandler(), session.getServiceConfig());
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionCreated(session); <-- called
> >> correctly
> >> session.getFilterChain().fireSessionOpened(session); <-- called
> >> correctly
> >>
> >> // Fire listener events.
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionCreated(session); <-- called correctly
> >> }
> >> }
> >>
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // logger.error("fireSessionDestroyed() : sessions == null < " +
> >> session);
> >> return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
> >> }
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after removal.
> >> if (sessions.isEmpty()) {
> >> lastSession = managedSessions.remove(serviceAddress, sessions);
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionClosed(session); <-- skipped
> >> sometimes
> >>
> >> // Fire listener events.
> >> try {
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionDestroyed(session); <-- skipped sometimes
> >> }
> >> } finally {
> >> // Fire a virtual service deactivation event for the last session of
> >> // the connector.
> >> // TODO double-check that this is *STILL* the last session. May not
> >> // be the case
> >> if (session.getService() instanceof IoConnector && lastSession) {
> >> fireServiceDeactivated(session.getService(),
> >> session.getServiceAddress(), session
> >> .getHandler(), session.getServiceConfig()); <-- skipped sometimes
> >> }
> >> }
> >> }
> >>
> >> the above fireSessionDestroyed method called as the same number as
> >> fireSessionCreated method.
> >>
> >> but the mysterious event is that the sessions is not gotten from
> >> managedSessions in fireSessionDestroyed,
> >> i.e. SocketAddress key is not found in managedSessions.
> >> then the remaining logic was skipped.
> >> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
> >>
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >>
> >> I think the reason that the 100 client threads tries to proceed "conect,
> >> send, receive, close",
> >> different sessions has the same client SocketAddress at times in server,
> >> and the origin of that problem is client's stress environment.
> >>
> >> It is possible the following senario :
> >>
> >> I/O processor thread I/O processor thread client thread client
> >> thread
> >> one two one two
> >> --------------------------------------------------------------------------------
> >> 1) connect(the same
> >> client SocketAddress)
> >> 2)fireSessionCreated
> >> 3)putIfAbsent
> >> 4)disconnect
> >>
> >> 5)connect(the same client SocketAddress)
> >> 6)fireSessionCreated
> >> 7)putIfAbsent
> >> 8)fireSessionDestroyed
> >> 9)remove
> >>
> >> 10)disconnect
> >> 11)fireSessionDestroyed
> >> 12)get fail
> >>
> >>
> >>
> >> I changed the mina source IoServiceListenerSupport.java
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> ...
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // return; <-- delete
> >> } else { // <-- add
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after removal.
> >> if (sessions.isEmpty()) {
> >> lastSession = managedSessions.remove(serviceAddress, sessions);
> >> }
> >> } // <-- add
> >> ....
> >>
> >> then the Create, Destroy call mismatch was cleared.
> >> but I don't know this is correct solution.
> >>
> >> Would you tell me this is correct or not?
> >>
> >> I think this situation is very rare but...
> >> I hope the mina is the best network framework
> >> and this situation shoud be also considered.
> >>
> >>
> >> Thank you
> >>
> >> J. H. Cha
> >
> >
> >
>
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: sessionClosed is not called sometimes in heavy stress test
contions.
Posted by jhcha <jh...@kds.co.kr>.
Thank Mike for replying my question.
I read your answers but...
I think MINA already knew the session was being closed in my questioned
situation.
because my server session always close after sending the echo data and
I debugged MINA source
org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method
always call doRemove() at (SocketIoProcessor.java:489 line).
then doRemove method always call
session.getServiceListeners().fireSessionDestroyed(session);
at (SocketIoProcessor.java:188 line).
try {
key.cancel();
ch.close();
} catch (IOException e) {
session.getFilterChain().fireExceptionCaught(session, e);
} finally {
releaseWriteBuffers(session);
session.getServiceListeners().fireSessionDestroyed(session); <---
SocketIoProcessor.java:188 line
}
and the above session parameter of the fireSessionDestroyed method was
always not null.
the orgin of the problem is the sessions having serviceAddress is not in
managedSessions hashmap
in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)
// Get the session set.
Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
IoServiceListenerSupport.java:192 line
// Ignore if unknown.
if (sessions == null) {
return;
}
The fireSessionDestroyed method call propagation stopped only sometimes
Because sessions is not gotten from managedSessions at
(IoServiceListenerSupport.java:192 line).
I used setIdleTimeout with read mode, but
if sessionClosed method is not called, sessionIdle or sessionClosed method
will never be called after that.
My question is that the filter chain fireSessionClosed method and listener'
sessionDestroyed method
Should not know if the destroying session be not in the managedSessions
hashmap ?
// Fire session events.
session.getFilterChain().fireSessionClosed(session); <---
IoServiceListenerSupport.java:208 line
// Fire listener events.
try {
for (IoServiceListener listener : listeners) {
listener.sessionDestroyed(session); <---
}
}
If fireSessionDestroyed method were called with the parameter the closing
session
but not in managedSessions hashmap, Should filter chain fireSessionClosed
method and
listener' sessionDestroyed method not called ?
in the following MINA source below the IoServiceListenerSupport.java:147
line.
public void fireSessionCreated(IoSession session) {
SocketAddress serviceAddress = session.getServiceAddress();
// Get the session set.
Set<IoSession> s = new IdentityHashSet<IoSession>();
Set<IoSession> sessions =
managedSessions.putIfAbsent(serviceAddress,
Collections.synchronizedSet(s));
boolean firstSession;
if (null == sessions) {
sessions = s;
firstSession = true;
} else {
firstSession = false;
}
// If already registered, ignore.
if (!sessions.add(session)) { <---
return;
}
// If the first connector session, fire a virtual service activation
event.
if (session.getService() instanceof IoConnector && firstSession) {
fireServiceActivated(session.getService(), session
.getServiceAddress(), session.getHandler(), session
.getServiceConfig());
}
// Fire session events.
session.getFilterChain().fireSessionCreated(session);
session.getFilterChain().fireSessionOpened(session);
// Fire listener events.
for (IoServiceListener listener : listeners) {
listener.sessionCreated(session);
}
}
I think the code "if (!sessions.add(session))" is always true,
So filter chain fireSessionCreated method and listener' sessionCreated
method is
always called in spite of the heavy unstable tcp stress condition.
I think
Creaing callback method and destroying callback method should have the same
calling count.
But creating callback logic and managedSessions hashmap are not related
directly
in the above fireSessionCreated method.
"not related directly" means that
whether sessions put managedSessions hashmap or not, fireSessionCreated and
sessionCreated are called.
Then destroying callback logic and managedSessions hashmap should be also
not related directly.
but MINA has the direct relation with managedSessions hashmap
"direct relation" means that
as the below code shows, if sessions is not found in managedSessions hashmap
then
fireSessionClosed and sessionDestroyed method are not called.
in the code "public void fireSessionDestroyed(IoSession session)" method
of IoServiceListenerSupport.java
...
// Get the session set.
Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
IoServiceListenerSupport.java:192 line
// Ignore if unknown.
if (sessions == null) { <---
return;
}
sessions.remove(session);
boolean lastSession = false;
// Try to remove the remaining empty session set after removal.
if (sessions.isEmpty()) {
lastSession = managedSessions.remove(serviceAddress, sessions);
<====
}
// Fire session events.
session.getFilterChain().fireSessionClosed(session);
// Fire listener events.
try {
for (IoServiceListener listener : listeners) {
listener.sessionDestroyed(session);
}
}
...
My inference is that
When heavy thread competition (and other session parameters but with the
same serviceAddress),
the one thread can remove the other using sessions from managedSessions
hashmap.
1) the one thread : sessions.isEmpty() succeeded in fireSessionDestroyed
2) the other thread : called fireSessionCreated
3) the other thread : sessions.add(session) suceeded in fireSessionCreated
4) the one thread : managedSessions.remove(serviceAddress, sessions);
(above "<===" line)
5) the other thread : called fireSessionDestroyed
6) the other thread : managedSessions.get(serviceAddress); fail
(above "<---" line)
other thread fails to get sessions from managedSessions hashmap with the
serviceAddress .
then fireSessionClosed propagation stops. (but should it continue!)
It is just my inference.
and it's possible for me to misunderstand the MINA logic.
Thank you again.
J. H. Cha
Mike Heath-4 wrote:
>
> It's possible that the server never received the TCP FIN packet and
> therefor did not know that it should close the IoSession. There are
> situations where a host has no way of knowing that the TCP socket has
> closed unless it tries to write data to the socket and that write fails.
> This is not a limitation of MINA. This is just the way socket I/O works.
>
> If this is the cause of your problem, you have a couple of options. You
> could use some type of watch dog mechanisms that sends data at a regular
> interval. If the connection is no longer valid, the write will fail and
> the socket will get closed thereby triggering a sessionClosed event. Or
> you could use the MINA read idle notification. This way after a certain
> amount of time has passed and your server hasn't received any data for a
> particular IoSession, you can manually close the IoSession.
>
> If your problem still persists after implementing one of these
> solutions, please let us know.
>
> -Mike
>
> jhcha wrote:
>> Dear Mina developers.
>> I'm trying to use mina 1.1.5 to develop the very big stress server.
>> I test my server with jmeter like some echo test now.
>>
>> test environment
>>
>> * jmeter :
>> echo message size : 1024 bytes
>> jmeter threads : 100
>> loop count : 1000
>> sampler : TCP Sampler
>> reuse connection : false
>>
>> * server : HP-UX 2 cores
>> java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
>> IA64, mixed mode)
>> my server(mina)
>>
>> * client : pc (dual core).
>> jmeter
>>
>> when I tested the stress, I found sessionCreated called counts are a
>> little
>> bigger than sessionClosed called counts.
>> so, I tried to find out why that mismatch occured during several days.
>> and I found the followings :
>>
>>
>> org.apache.mina.common.support.IoServiceListenerSupport
>> {
>>
>> public void fireSessionCreated(IoSession session) {
>> SocketAddress serviceAddress = session.getServiceAddress();
>>
>> // Get the session set.
>> Set<IoSession> s = new IdentityHashSet<IoSession>();
>> Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
>> Collections
>> .synchronizedSet(s));
>> boolean firstSession;
>>
>> if (null == sessions) {
>> sessions = s;
>> firstSession = true;
>> } else {
>> firstSession = false;
>> }
>>
>> // If already registered, ignore.
>> if (!sessions.add(session)) {
>> return;
>> }
>>
>> // If the first connector session, fire a virtual service activation
>> // event.
>> if (session.getService() instanceof IoConnector && firstSession) {
>> fireServiceActivated(session.getService(),
>> session.getServiceAddress(),
>> session
>> .getHandler(), session.getServiceConfig());
>> }
>>
>> // Fire session events.
>> session.getFilterChain().fireSessionCreated(session); <-- called
>> correctly
>> session.getFilterChain().fireSessionOpened(session); <-- called
>> correctly
>>
>> // Fire listener events.
>> for (IoServiceListener listener : listeners) {
>> listener.sessionCreated(session); <-- called correctly
>> }
>> }
>>
>>
>> public void fireSessionDestroyed(IoSession session) {
>> boolean lastSession = false;
>> SocketAddress serviceAddress = session.getServiceAddress();
>>
>> // Get the session set.
>> Set<IoSession> sessions = managedSessions.get(serviceAddress);
>> // Ignore if unknown.
>> if (sessions == null) {
>> // logger.error("fireSessionDestroyed() : sessions == null < " +
>> session);
>> return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
>> }
>>
>> sessions.remove(session);
>>
>> // Try to remove the remaining empty session set after removal.
>> if (sessions.isEmpty()) {
>> lastSession = managedSessions.remove(serviceAddress, sessions);
>> }
>>
>> // Fire session events.
>> session.getFilterChain().fireSessionClosed(session); <-- skipped
>> sometimes
>>
>> // Fire listener events.
>> try {
>> for (IoServiceListener listener : listeners) {
>> listener.sessionDestroyed(session); <-- skipped sometimes
>> }
>> } finally {
>> // Fire a virtual service deactivation event for the last session of
>> // the connector.
>> // TODO double-check that this is *STILL* the last session. May not
>> // be the case
>> if (session.getService() instanceof IoConnector && lastSession) {
>> fireServiceDeactivated(session.getService(),
>> session.getServiceAddress(), session
>> .getHandler(), session.getServiceConfig()); <-- skipped sometimes
>> }
>> }
>> }
>>
>> the above fireSessionDestroyed method called as the same number as
>> fireSessionCreated method.
>>
>> but the mysterious event is that the sessions is not gotten from
>> managedSessions in fireSessionDestroyed,
>> i.e. SocketAddress key is not found in managedSessions.
>> then the remaining logic was skipped.
>> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
>>
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>>
>> I think the reason that the 100 client threads tries to proceed "conect,
>> send, receive, close",
>> different sessions has the same client SocketAddress at times in server,
>> and the origin of that problem is client's stress environment.
>>
>> It is possible the following senario :
>>
>> I/O processor thread I/O processor thread client thread client
>> thread
>> one two one two
>> --------------------------------------------------------------------------------
>> 1) connect(the same
>> client SocketAddress)
>> 2)fireSessionCreated
>> 3)putIfAbsent
>> 4)disconnect
>>
>> 5)connect(the same client SocketAddress)
>> 6)fireSessionCreated
>> 7)putIfAbsent
>> 8)fireSessionDestroyed
>> 9)remove
>>
>> 10)disconnect
>> 11)fireSessionDestroyed
>> 12)get fail
>>
>>
>>
>> I changed the mina source IoServiceListenerSupport.java
>>
>> org.apache.mina.common.support.IoServiceListenerSupport
>> ...
>>
>> public void fireSessionDestroyed(IoSession session) {
>> boolean lastSession = false;
>> SocketAddress serviceAddress = session.getServiceAddress();
>>
>> // Get the session set.
>> Set<IoSession> sessions = managedSessions.get(serviceAddress);
>> // Ignore if unknown.
>> if (sessions == null) {
>> // return; <-- delete
>> } else { // <-- add
>>
>> sessions.remove(session);
>>
>> // Try to remove the remaining empty session set after removal.
>> if (sessions.isEmpty()) {
>> lastSession = managedSessions.remove(serviceAddress, sessions);
>> }
>> } // <-- add
>> ....
>>
>> then the Create, Destroy call mismatch was cleared.
>> but I don't know this is correct solution.
>>
>> Would you tell me this is correct or not?
>>
>> I think this situation is very rare but...
>> I hope the mina is the best network framework
>> and this situation shoud be also considered.
>>
>>
>> Thank you
>>
>> J. H. Cha
>
>
>
--
View this message in context: http://www.nabble.com/sessionClosed-is-not-called-sometimes-in-heavy-stress-test-contions.-tp15265671s16868p15350882.html
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
Re: sessionClosed is not called sometimes in heavy stress test contions.
Posted by Mike Heath <mh...@apache.org>.
It's possible that the server never received the TCP FIN packet and
therefor did not know that it should close the IoSession. There are
situations where a host has no way of knowing that the TCP socket has
closed unless it tries to write data to the socket and that write fails.
This is not a limitation of MINA. This is just the way socket I/O works.
If this is the cause of your problem, you have a couple of options. You
could use some type of watch dog mechanisms that sends data at a regular
interval. If the connection is no longer valid, the write will fail and
the socket will get closed thereby triggering a sessionClosed event. Or
you could use the MINA read idle notification. This way after a certain
amount of time has passed and your server hasn't received any data for a
particular IoSession, you can manually close the IoSession.
If your problem still persists after implementing one of these
solutions, please let us know.
-Mike
jhcha wrote:
> Dear Mina developers.
> I'm trying to use mina 1.1.5 to develop the very big stress server.
> I test my server with jmeter like some echo test now.
>
> test environment
>
> * jmeter :
> echo message size : 1024 bytes
> jmeter threads : 100
> loop count : 1000
> sampler : TCP Sampler
> reuse connection : false
>
> * server : HP-UX 2 cores
> java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> IA64, mixed mode)
> my server(mina)
>
> * client : pc (dual core).
> jmeter
>
> when I tested the stress, I found sessionCreated called counts are a little
> bigger than sessionClosed called counts.
> so, I tried to find out why that mismatch occured during several days.
> and I found the followings :
>
>
> org.apache.mina.common.support.IoServiceListenerSupport
> {
>
> public void fireSessionCreated(IoSession session) {
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> s = new IdentityHashSet<IoSession>();
> Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
> Collections
> .synchronizedSet(s));
> boolean firstSession;
>
> if (null == sessions) {
> sessions = s;
> firstSession = true;
> } else {
> firstSession = false;
> }
>
> // If already registered, ignore.
> if (!sessions.add(session)) {
> return;
> }
>
> // If the first connector session, fire a virtual service activation
> // event.
> if (session.getService() instanceof IoConnector && firstSession) {
> fireServiceActivated(session.getService(), session.getServiceAddress(),
> session
> .getHandler(), session.getServiceConfig());
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionCreated(session); <-- called
> correctly
> session.getFilterChain().fireSessionOpened(session); <-- called correctly
>
> // Fire listener events.
> for (IoServiceListener listener : listeners) {
> listener.sessionCreated(session); <-- called correctly
> }
> }
>
>
> public void fireSessionDestroyed(IoSession session) {
> boolean lastSession = false;
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> // Ignore if unknown.
> if (sessions == null) {
> // logger.error("fireSessionDestroyed() : sessions == null < " +
> session);
> return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
> }
>
> sessions.remove(session);
>
> // Try to remove the remaining empty session set after removal.
> if (sessions.isEmpty()) {
> lastSession = managedSessions.remove(serviceAddress, sessions);
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session); <-- skipped sometimes
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session); <-- skipped sometimes
> }
> } finally {
> // Fire a virtual service deactivation event for the last session of
> // the connector.
> // TODO double-check that this is *STILL* the last session. May not
> // be the case
> if (session.getService() instanceof IoConnector && lastSession) {
> fireServiceDeactivated(session.getService(),
> session.getServiceAddress(), session
> .getHandler(), session.getServiceConfig()); <-- skipped sometimes
> }
> }
> }
>
> the above fireSessionDestroyed method called as the same number as
> fireSessionCreated method.
>
> but the mysterious event is that the sessions is not gotten from
> managedSessions in fireSessionDestroyed,
> i.e. SocketAddress key is not found in managedSessions.
> then the remaining logic was skipped.
> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
>
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
>
> I think the reason that the 100 client threads tries to proceed "conect,
> send, receive, close",
> different sessions has the same client SocketAddress at times in server,
> and the origin of that problem is client's stress environment.
>
> It is possible the following senario :
>
> I/O processor thread I/O processor thread client thread client
> thread
> one two one two
> --------------------------------------------------------------------------------
> 1) connect(the same
> client SocketAddress)
> 2)fireSessionCreated
> 3)putIfAbsent
> 4)disconnect
>
> 5)connect(the same client SocketAddress)
> 6)fireSessionCreated
> 7)putIfAbsent
> 8)fireSessionDestroyed
> 9)remove
>
> 10)disconnect
> 11)fireSessionDestroyed
> 12)get fail
>
>
>
> I changed the mina source IoServiceListenerSupport.java
>
> org.apache.mina.common.support.IoServiceListenerSupport
> ...
>
> public void fireSessionDestroyed(IoSession session) {
> boolean lastSession = false;
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> // Ignore if unknown.
> if (sessions == null) {
> // return; <-- delete
> } else { // <-- add
>
> sessions.remove(session);
>
> // Try to remove the remaining empty session set after removal.
> if (sessions.isEmpty()) {
> lastSession = managedSessions.remove(serviceAddress, sessions);
> }
> } // <-- add
> ....
>
> then the Create, Destroy call mismatch was cleared.
> but I don't know this is correct solution.
>
> Would you tell me this is correct or not?
>
> I think this situation is very rare but...
> I hope the mina is the best network framework
> and this situation shoud be also considered.
>
>
> Thank you
>
> J. H. Cha