You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Francesco Vivoli <f....@gmail.com> on 2011/02/08 10:22:37 UTC
Weird connection pattern
Hi there
I'm still in the process of assessing an application which make a fairly
heavy use of mina. It is packaged as a webapp deployed under tomcat.
Using netstat I have noticed a strange (to me) connection pattern like the
one attached below. I understand that this is very application specific and
that it's hard to draw conclusions just looking at this trace, but this
application itself doesn't open internal connections this way: it is
basically a bunch of acceptors and connectors, so nothing that would explain
a netstat output like this. Moreover, I have used a sniffer to inspect
traffic but I haven't captured any of it. Strange.
Does anybody have ever noticed something like this? The server running this
application has shown to sometimes to run out of resources (not being able from
the system nonpaged pool and thus not being able to create any more
connections) and I'm trying to get to the cause. I have been monitoring the
trend of these connections and they seem to enlarge and shrink but
definitely they show a steady growth. Still, I don't know who's opening
them. I hope you guys can help:)
Btw, production is still running 2.0.0-M6.
Thanks a lot, regards
Francesco
-----
netsat output
TCP 127.0.0.1:1064 127.0.0.1:1065 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1065 127.0.0.1:1064 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1066 127.0.0.1:1067 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1067 127.0.0.1:1066 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1068 127.0.0.1:1069 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1069 127.0.0.1:1068 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1070 127.0.0.1:1071 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1071 127.0.0.1:1070 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1073 127.0.0.1:1074 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1074 127.0.0.1:1073 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1075 127.0.0.1:1076 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1076 127.0.0.1:1075 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1077 127.0.0.1:1078 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1078 127.0.0.1:1077 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1079 127.0.0.1:1080 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1080 127.0.0.1:1079 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1081 127.0.0.1:1082 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1082 127.0.0.1:1081 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1083 127.0.0.1:1084 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1084 127.0.0.1:1083 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1085 127.0.0.1:1086 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1086 127.0.0.1:1085 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1087 127.0.0.1:1088 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1088 127.0.0.1:1087 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1090 127.0.0.1:1091 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1091 127.0.0.1:1090 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1092 127.0.0.1:1093 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1093 127.0.0.1:1092 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1094 127.0.0.1:1095 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1095 127.0.0.1:1094 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1096 127.0.0.1:1097 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1097 127.0.0.1:1096 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1098 127.0.0.1:1099 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1099 127.0.0.1:1098 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1101 127.0.0.1:1102 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1102 127.0.0.1:1101 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1103 127.0.0.1:1104 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1104 127.0.0.1:1103 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1105 127.0.0.1:1106 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1106 127.0.0.1:1105 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1107 127.0.0.1:1108 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1108 127.0.0.1:1107 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1109 127.0.0.1:1110 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1110 127.0.0.1:1109 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1112 127.0.0.1:1113 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1113 127.0.0.1:1112 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1114 127.0.0.1:1115 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1115 127.0.0.1:1114 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1116 127.0.0.1:1117 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1117 127.0.0.1:1116 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1118 127.0.0.1:1119 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1119 127.0.0.1:1118 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1120 127.0.0.1:1121 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1121 127.0.0.1:1120 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1123 127.0.0.1:1124 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1124 127.0.0.1:1123 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1125 127.0.0.1:1126 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1126 127.0.0.1:1125 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1127 127.0.0.1:1128 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1128 127.0.0.1:1127 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1129 127.0.0.1:1130 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1130 127.0.0.1:1129 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1131 127.0.0.1:1132 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1132 127.0.0.1:1131 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1134 127.0.0.1:1135 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1135 127.0.0.1:1134 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1136 127.0.0.1:1137 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1137 127.0.0.1:1136 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1138 127.0.0.1:1139 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1139 127.0.0.1:1138 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1140 127.0.0.1:1141 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1141 127.0.0.1:1140 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1142 127.0.0.1:1143 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1143 127.0.0.1:1142 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1144 127.0.0.1:1145 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1145 127.0.0.1:1144 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1146 127.0.0.1:1147 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1147 127.0.0.1:1146 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1148 127.0.0.1:1149 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1149 127.0.0.1:1148 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1150 127.0.0.1:1151 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1151 127.0.0.1:1150 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1153 127.0.0.1:1154 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1154 127.0.0.1:1153 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1155 127.0.0.1:1156 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1156 127.0.0.1:1155 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1157 127.0.0.1:1158 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1158 127.0.0.1:1157 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1159 127.0.0.1:1160 ESTABLISHED 2792
[tomcat6.exe]
TCP 127.0.0.1:1160 127.0.0.1:1159 ESTABLISHED 2792
[tomcat6.exe]
Re: Weird connection pattern
Posted by Francesco Vivoli <f....@gmail.com>.
I'm reposting this, I had left out the mina-users ml address
On Thu, Feb 10, 2011 at 2:06 PM, Francesco Vivoli <f....@gmail.com>wrote:
>
>
> On Thu, Feb 10, 2011 at 1:29 PM, Emmanuel Lécharny <el...@apache.org>wrote:
>
>> On 2/10/11 12:57 PM, Francesco Vivoli wrote:
>>
>>> Ok, I have run some other tests with mina-core-2.0.2 and this connection
>>> pattern is still there... It surely has to deal with Windows since, on
>>> Linux
>>> and Mac OS X there's no sign of it.
>>>
>>> I am still trying to capture traffic going through any of these
>>> connection
>>> but it can't see any of it.
>>>
>>
>> Are you using Wireshark ?
>>
>>
> Yes, why?
>
>
>> Frankly I'm a bit concerned as we had a system failure weeks ago where
>>> Windows reported it was out of nonpaged pool, and it stopped being able
>>> to
>>> create new connections. On average netstat shows around 250-300
>>> established
>>> connections of which some 80 are these "unknown" ones. It also has 30-40
>>> TIME_WAIT ones.
>>>
>>
>> 300 established connections is just nothing. On a loaded server, you may
>> get hundreds of thousands established connections. Although I would never
>> run such a load on a windows machine, hat's for sure ...
>>
>>
> I know it's not much, that's why it bothers me having run out of pool space
> to allocate new connections...
>
>
>
>> Since the app we're working on basically is just a webapp using mina and
>>> a
>>> DB, and being able to pinpoint db connections easily, do you have some
>>> ideas
>>> about what could be causing this and if we should worry about it (right
>>> now
>>> we do, maybe it's a well known Windows/NIO behavior that I haven't been
>>> able
>>> to locate).
>>>
>> Question is : can you reproduce this behaviour on a test machine with few
>> connected sessions? It would be easier to determinate what can cause those
>> localhost connections to be opened.
>>
>>
> This is going to be my next step:)
>
>
>> Also I'm not sue that Process Explorer can help in finding which process
>> exactly is creating those connections (Process Explorer is a bit like lsof
>> on linux).
>>
>>
> netstat shows that they are all owned by the tomcat process.
>
> Thanks, regards
> Francesco
>
>
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>>
>
Re: Weird connection pattern
Posted by Francesco Vivoli <f....@gmail.com>.
Ok, I have run some other tests with mina-core-2.0.2 and this connection
pattern is still there... It surely has to deal with Windows since, on Linux
and Mac OS X there's no sign of it.
I am still trying to capture traffic going through any of these connection
but it can't see any of it.
Frankly I'm a bit concerned as we had a system failure weeks ago where
Windows reported it was out of nonpaged pool, and it stopped being able to
create new connections. On average netstat shows around 250-300 established
connections of which some 80 are these "unknown" ones. It also has 30-40
TIME_WAIT ones.
Since the app we're working on basically is just a webapp using mina and a
DB, and being able to pinpoint db connections easily, do you have some ideas
about what could be causing this and if we should worry about it (right now
we do, maybe it's a well known Windows/NIO behavior that I haven't been able
to locate).
Thanks again, regards
Francesco
On Tue, Feb 8, 2011 at 11:33 AM, Francesco Vivoli <f....@gmail.com>wrote:
> we have migrated to 2.0.2 in our dev environment, I still have to run tests
> in pre-production though. I'll follow up once I have done so and checked
> whether this behavior persists.
>
> regards
> F
>
>
> On Tue, Feb 8, 2011 at 10:29 AM, Emmanuel Lecharny <el...@gmail.com>wrote:
>
>> On 2/8/11 10:22 AM, Francesco Vivoli wrote:
>>
>>> Hi there
>>>
>>> I'm still in the process of assessing an application which make a fairly
>>> heavy use of mina. It is packaged as a webapp deployed under tomcat.
>>>
>>> Using netstat I have noticed a strange (to me) connection pattern like
>>> the
>>> one attached below. I understand that this is very application specific
>>> and
>>> that it's hard to draw conclusions just looking at this trace, but this
>>> application itself doesn't open internal connections this way: it is
>>> basically a bunch of acceptors and connectors, so nothing that would
>>> explain
>>> a netstat output like this. Moreover, I have used a sniffer to inspect
>>> traffic but I haven't captured any of it. Strange.
>>>
>>> Does anybody have ever noticed something like this? The server running
>>> this
>>> application has shown to sometimes to run out of resources (not being
>>> able from
>>> the system nonpaged pool and thus not being able to create any more
>>> connections) and I'm trying to get to the cause. I have been monitoring
>>> the
>>> trend of these connections and they seem to enlarge and shrink but
>>> definitely they show a steady growth. Still, I don't know who's opening
>>> them. I hope you guys can help:)
>>>
>>> Btw, production is still running 2.0.0-M6.
>>>
>>
>> First thing I would do would be to switch to MINA 2.0.2. 2.0.0-M6 is just
>> totally outdated.
>>
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>>
>
Re: Weird connection pattern
Posted by Francesco Vivoli <f....@gmail.com>.
we have migrated to 2.0.2 in our dev environment, I still have to run tests
in pre-production though. I'll follow up once I have done so and checked
whether this behavior persists.
regards
F
On Tue, Feb 8, 2011 at 10:29 AM, Emmanuel Lecharny <el...@gmail.com>wrote:
> On 2/8/11 10:22 AM, Francesco Vivoli wrote:
>
>> Hi there
>>
>> I'm still in the process of assessing an application which make a fairly
>> heavy use of mina. It is packaged as a webapp deployed under tomcat.
>>
>> Using netstat I have noticed a strange (to me) connection pattern like the
>> one attached below. I understand that this is very application specific
>> and
>> that it's hard to draw conclusions just looking at this trace, but this
>> application itself doesn't open internal connections this way: it is
>> basically a bunch of acceptors and connectors, so nothing that would
>> explain
>> a netstat output like this. Moreover, I have used a sniffer to inspect
>> traffic but I haven't captured any of it. Strange.
>>
>> Does anybody have ever noticed something like this? The server running
>> this
>> application has shown to sometimes to run out of resources (not being able
>> from
>> the system nonpaged pool and thus not being able to create any more
>> connections) and I'm trying to get to the cause. I have been monitoring
>> the
>> trend of these connections and they seem to enlarge and shrink but
>> definitely they show a steady growth. Still, I don't know who's opening
>> them. I hope you guys can help:)
>>
>> Btw, production is still running 2.0.0-M6.
>>
>
> First thing I would do would be to switch to MINA 2.0.2. 2.0.0-M6 is just
> totally outdated.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: Weird connection pattern
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 2/8/11 10:22 AM, Francesco Vivoli wrote:
> Hi there
>
> I'm still in the process of assessing an application which make a fairly
> heavy use of mina. It is packaged as a webapp deployed under tomcat.
>
> Using netstat I have noticed a strange (to me) connection pattern like the
> one attached below. I understand that this is very application specific and
> that it's hard to draw conclusions just looking at this trace, but this
> application itself doesn't open internal connections this way: it is
> basically a bunch of acceptors and connectors, so nothing that would explain
> a netstat output like this. Moreover, I have used a sniffer to inspect
> traffic but I haven't captured any of it. Strange.
>
> Does anybody have ever noticed something like this? The server running this
> application has shown to sometimes to run out of resources (not being able from
> the system nonpaged pool and thus not being able to create any more
> connections) and I'm trying to get to the cause. I have been monitoring the
> trend of these connections and they seem to enlarge and shrink but
> definitely they show a steady growth. Still, I don't know who's opening
> them. I hope you guys can help:)
>
> Btw, production is still running 2.0.0-M6.
First thing I would do would be to switch to MINA 2.0.2. 2.0.0-M6 is
just totally outdated.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com