You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Trustin Lee <tr...@gmail.com> on 2005/07/13 03:35:23 UTC
Re: losing packets when load testing my mina app
Hi Alex,
2005/7/13, Alex Burmester <ad...@plushpix.com>:
>
> Hi Trustin, I have built a protocol router on top of mina
> and it generally runs very well.
Great.
I have found that when I fire up about 100 threads and pound on it with my
> load testing client that I do get a few lost packets that never make it to
> the mina server process. I did a tcpdump and found that the server
> machine is issuing tcp resets on the packets that are not showing up.
> Not sure why yet. I assume that it's some tcp parameters I need to bump
> up on the linux box but I thought I would check with you to see if there
> are any tunable parameters in the mina framework or in the jvm that you
> know of that I should check.
Well, are you using MINA 0.7.3? Please try to upgrade to 0.7.3 if you're
using an older version. You can configure all socket parameters by calling
IoSession.getConfig() and downcasting the returned object into
SocketSessionConfig. To configure ServerSocketChannel, you'll have to
upgrade to 0.9 for now.
Please let me know the result after you upgrade to 0.7.3.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: losing packets when load testing my mina app
Posted by Alex Burmester <ad...@plushpix.com>.
I think I have isolated the problem to a firewall
between the two machines I am using. If I look at traffic dumps from both
the client and the server, I see the resets as originating from the client
when I look at the server's dump but when I look at the client dump I see
no resets, I just see no acknowledgement of the packet so the client does
a few tcp retransmissions of the packet which the server then ignores
because of the reset.
This was confirmed by running the load test client on another machine on
the same subnet and no resets were triggered.
Thanks for helpful pointers all.
Alex.
On Thu, 14 Jul 2005, Vinod Panicker wrote:
> On 7/14/05, Alex Burmester <ad...@plushpix.com> wrote:
> > I just noticed in my ethereal traffic that it is the load testing client
> > that is initiating the tcp reset packets so this may be unrelated to mina
> > as the client currently doesn't use mina.
>
> IIRC, the RST would be generated if the client got an unexpected
> response/timeout from the server.
>
> > The server is runing mina 0.7.3 using jdk1.5.0_04 and the os/kernel is
> > redhat 2.4.21-20.ELsmp
>
> Linux kernels 2.6.x are known to have an improved tcp implementation -
> maybe that would help.
>
> > The client is fedora 2.6.10-1.741_FC3smp
> >
> > What is strange is that I don't see any tcp errors on either machine
> > using netstat -i eth0 -t -c 10
> >
> > Both jvms have ample memory allocated.
>
> Could you tell me the command line args that you are using for the
> jvm? And did you notice the memory/cpu utilization of both apps
> during the test?
>
> > After running the load test for about 10 seconds I start to see occasional
> > read timeouts on the client. First I thought the mina server was not
> > responding but then I found that the client is actually issuing a tcp
> > reset before actually trying to send it's payload to the mina server.
> > As a result the mina server never actually accepts the packet and the
> > client after doing a tcp retransmission a few times eventually times out.
> >
> > Any suggestions?
>
> You could post the source and I could try it out on my machines and
> see if the problem can be reproduced.
>
> --snip--
>
> Regards,
> Vinod.
>
Re: losing packets when load testing my mina app
Posted by Vinod Panicker <vi...@gmail.com>.
On 7/14/05, Alex Burmester <ad...@plushpix.com> wrote:
> I just noticed in my ethereal traffic that it is the load testing client
> that is initiating the tcp reset packets so this may be unrelated to mina
> as the client currently doesn't use mina.
IIRC, the RST would be generated if the client got an unexpected
response/timeout from the server.
> The server is runing mina 0.7.3 using jdk1.5.0_04 and the os/kernel is
> redhat 2.4.21-20.ELsmp
Linux kernels 2.6.x are known to have an improved tcp implementation -
maybe that would help.
> The client is fedora 2.6.10-1.741_FC3smp
>
> What is strange is that I don't see any tcp errors on either machine
> using netstat -i eth0 -t -c 10
>
> Both jvms have ample memory allocated.
Could you tell me the command line args that you are using for the
jvm? And did you notice the memory/cpu utilization of both apps
during the test?
> After running the load test for about 10 seconds I start to see occasional
> read timeouts on the client. First I thought the mina server was not
> responding but then I found that the client is actually issuing a tcp
> reset before actually trying to send it's payload to the mina server.
> As a result the mina server never actually accepts the packet and the
> client after doing a tcp retransmission a few times eventually times out.
>
> Any suggestions?
You could post the source and I could try it out on my machines and
see if the problem can be reproduced.
--snip--
Regards,
Vinod.
Re: losing packets when load testing my mina app
Posted by Alex Burmester <ad...@plushpix.com>.
I just noticed in my ethereal traffic that it is the load testing client
that is initiating the tcp reset packets so this may be unrelated to mina
as the client currently doesn't use mina.
The server is runing mina 0.7.3 using jdk1.5.0_04 and the os/kernel is
redhat 2.4.21-20.ELsmp
The client is fedora 2.6.10-1.741_FC3smp
What is strange is that I don't see any tcp errors on either machine
using netstat -i eth0 -t -c 10
Both jvms have ample memory allocated.
After running the load test for about 10 seconds I start to see occasional
read timeouts on the client. First I thought the mina server was not
responding but then I found that the client is actually issuing a tcp
reset before actually trying to send it's payload to the mina server.
As a result the mina server never actually accepts the packet and the
client after doing a tcp retransmission a few times eventually times out.
Any suggestions?
Here is a sample of what I'm seeing on the wire:
client 10.1.36.45
mina server 206.246.214.14
No. Time Source Destination Protocol
Info
36334 36.252292 10.1.36.45 206.246.213.14 TCP
41622 > 11111 [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=607287694
TSER=0 WS=2
36335 36.252323 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=262940360
TSER=607287694 WS=0
36336 36.252583 10.1.36.45 206.246.213.14 TCP
41622 > 11111 [RST] Seq=1 Ack=3719067569 Win=23168 Len=0
36337 36.252669 10.1.36.45 206.246.213.14 TCP
41622 > 11111 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=607287694
TSER=262940360
36338 36.252681 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
36341 36.253272 10.1.36.45 206.246.213.14 TCP
41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607287695
TSER=262940360
36342 36.253286 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
36683 36.458444 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131
TSV=607287900 TSER=262940360
36684 36.458469 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
37496 36.868383 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131
TSV=607288310 TSER=262940360
37497 36.868405 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
39146 37.688160 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131
TSV=607289130 TSER=262940360
39147 37.688178 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
42427 39.328107 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131
TSV=607290770 TSER=262940360
42428 39.328171 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
47930 42.607621 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131
TSV=607294050 TSER=262940360
47931 42.607651 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
51244 44.795623 10.1.36.45 206.246.213.14 TCP
41622 > 11111 [FIN, ACK] Seq=132 Ack=1 Win=5840 Len=0 TSV=607296239
TSER=262940360
51245 44.795631 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
59141 49.165337 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840
Len=131 TSV=607300610 TSER=262940360
59142 49.165362 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
80975 62.283799 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840
Len=131 TSV=607313730 TSER=262940360
80976 62.283822 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
102661 88.518412 10.1.36.45 206.246.213.14 TCP
[TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840
Len=131 TSV=607339970 TSER=262940360
102662 88.518428 206.246.213.14 10.1.36.45 TCP
11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0
Thanks,
Alex.
On Wed, 13 Jul 2005, Vinod Panicker wrote:
> During load testing, I've also experienced connection resets and
> drops, but they disappeared when I gave the JVM more memory to play
> with. Search in the archives for this (-Xms and something else i
> forget).
>
> Regards,
> Vinod.
>
> On 7/13/05, Trustin Lee <tr...@gmail.com> wrote:
> > Hi Alex,
> >
> >
> > 2005/7/13, Alex Burmester <ad...@plushpix.com>:
> > > Hi Trustin, I have built a protocol router on top of mina
> > > and it generally runs very well.
> >
> >
> > Great.
> >
> > > I have found that when I fire up about 100 threads and pound on it with my
> > > load testing client that I do get a few lost packets that never make it to
> > > the mina server process. I did a tcpdump and found that the server
> > > machine is issuing tcp resets on the packets that are not showing up.
> > > Not sure why yet. I assume that it's some tcp parameters I need to bump
> > > up on the linux box but I thought I would check with you to see if there
> > > are any tunable parameters in the mina framework or in the jvm that you
> > > know of that I should check.
> >
> >
> > Well, are you using MINA 0.7.3? Please try to upgrade to 0.7.3 if you're
> > using an older version. You can configure all socket parameters by calling
> > IoSession.getConfig() and downcasting the returned object into
> > SocketSessionConfig. To configure ServerSocketChannel, you'll have to
> > upgrade to 0.9 for now.
> >
> > Please let me know the result after you upgrade to 0.7.3.
> >
> > Trustin--
> > what we call human nature is actually human habit
> > --
> > http://gleamynode.net/
>
Re: losing packets when load testing my mina app
Posted by Vinod Panicker <vi...@gmail.com>.
During load testing, I've also experienced connection resets and
drops, but they disappeared when I gave the JVM more memory to play
with. Search in the archives for this (-Xms and something else i
forget).
Regards,
Vinod.
On 7/13/05, Trustin Lee <tr...@gmail.com> wrote:
> Hi Alex,
>
>
> 2005/7/13, Alex Burmester <ad...@plushpix.com>:
> > Hi Trustin, I have built a protocol router on top of mina
> > and it generally runs very well.
>
>
> Great.
>
> > I have found that when I fire up about 100 threads and pound on it with my
> > load testing client that I do get a few lost packets that never make it to
> > the mina server process. I did a tcpdump and found that the server
> > machine is issuing tcp resets on the packets that are not showing up.
> > Not sure why yet. I assume that it's some tcp parameters I need to bump
> > up on the linux box but I thought I would check with you to see if there
> > are any tunable parameters in the mina framework or in the jvm that you
> > know of that I should check.
>
>
> Well, are you using MINA 0.7.3? Please try to upgrade to 0.7.3 if you're
> using an older version. You can configure all socket parameters by calling
> IoSession.getConfig() and downcasting the returned object into
> SocketSessionConfig. To configure ServerSocketChannel, you'll have to
> upgrade to 0.9 for now.
>
> Please let me know the result after you upgrade to 0.7.3.
>
> Trustin--
> what we call human nature is actually human habit
> --
> http://gleamynode.net/