You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by "Huesgen, Chad" <ch...@litle.com> on 2008/08/28 17:35:45 UTC
Question about Scalability of Jmeter-Server
Hello All,
I am designing some performance tests using jmeter-server and am getting
some inconsistent results. My base assumption is that 2 jmeter servers
will get better performance numbers then 1 jmeter server or 1 jmeter
client by spreading the load across. I am running a very simple ping
test here basically hitting a Tomcat server with a helloworld.jsp with a
500 milli sleep in it. The setup is I have 2 servers acting as jmeter
servers and one server that is hosting Tomcat and the client. The
servers only have one hop to each other and all tests run at night so no
network traffic should affect the ping. My question is have people seen
numbers better and more consistent by spreading jmeter clients out
across 3 servers and not using jmeter-server? If I saw a Std Dev all
under 100 and close to each other for each run than it would make me
feel much better with the results and confident that I can move onto
designing the real perf tests for the app.
Here are some numbers that I am seeing:
Test Case Standard Dev Avg Median Max Min 90ile
95ile 99ile
Jmeter Only 400 Threads 181.0445542 524.6929 503 9507
501 504 516 1223
Jmeter Only 400 Threads 126.5504985 517.8482 503 3508
501 504 506 965
Jmeter Only 400 Threads 213.6036281 534.2436 503 3517
501 504 516 1454
One Server 400 Threads 203.1225482 537.1209 504 3577
501 558 582 1208
One Server 400 Threads 211.2885383 545.0142 504 3582
501 569 617 1295
One Server 400 Threads 238.3708453 539.7173 504 9504
501 559 582 1279
Two Servers 400 Threads 167.2694959 521.1874 503 5060
501 504 505 1422
Two Servers 400 Threads 175.2300215 523.661 503 3508 501
504 506 1310
Two Servers 400 Threads 187.0677548 521.4626 503 4640
501 504 505 1142
Jmeter Only 500 Threads 388.3772604 559.2594 503 9509
501 507 594 2402
Jmeter Only 500 Threads 346.868954 549.8935 503 9509
501 507 672 1475
Jmeter Only 500 Threads 340.8531506 554.2277 503 10191
501 506 633 1782
One Server 500 Threads 499.4419659 545.5399 504 21513
501 560 576 1095
One Server 500 Threads 230.4885537 546.537 504 3595 501
564 578 1752
One Server 500 Threads 274.5812614 551.7715 505 5331
501 564 579 1442
Two Servers 500 Threads 242.6634829 538.9398 503 9503
501 504 524 1775
Two Servers 500 Threads 177.9699861 524.772 503 4398 501
504 514 1444
Two Servers 500 Threads 365.8320039 540.0532 503 22621
501 505 525 1587
Thanks,
Chad
NOTICE: This message, including all attachments transmitted with it, is for the use of the addressee only. It may contain proprietary, confidential and/or legally privileged information belonging to Litle & Co. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient, you must not, directly or indirectly, use, disclose, distribute, print or copy any part of this message. If you believe you have received this message in error, please delete it and all copies of it from your system and notify the sender immediately by reply e-mail. Thank you.
Re: Question about Scalability of Jmeter-Server
Posted by Peter Lin <wo...@gmail.com>.
On Thu, Aug 28, 2008 at 12:15 PM, Huesgen, Chad <ch...@litle.com> wrote:
> Sorry should of put that in there. I am running all tests in non-gui
> mode, gave jmeter-server 1 gig of mem and cycling jmeter-servers between
> each test. I moved the client to the webserver to reduce the amount of
> network hops that the client and server have to make. Since this is a
> ping test, basically display hello world, I would definitely expect the
> min and median to be very consistent but the max and 9xile's I expected
> to be closer to the median. I would definitely expect the client to be
> a little of a bottleneck since 2 servers are responding back to it which
> is why I was curious to see if people used multiple jmeter-clients
> instead of jmeter-servers since that would get rid of the multiple hits
> to one client. I tried to minimize the bottleneck of the client by
> running it in non-gui mode and also using a simple datawriter to write
> my results.
>
> Thanks,
>
> Chad
>
the percentile statistics can vary for a variety of reasons. One thing
you can do is look at the network usage on the system running the
webserver.
if the webserver's network IO is maxed out, it can affect those
statistics. You might want to look at what percent of the requests are
between the mean and 90%.
peter
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
RE: Question about Scalability of Jmeter-Server
Posted by "Huesgen, Chad" <ch...@litle.com>.
Sorry should of put that in there. I am running all tests in non-gui
mode, gave jmeter-server 1 gig of mem and cycling jmeter-servers between
each test. I moved the client to the webserver to reduce the amount of
network hops that the client and server have to make. Since this is a
ping test, basically display hello world, I would definitely expect the
min and median to be very consistent but the max and 9xile's I expected
to be closer to the median. I would definitely expect the client to be
a little of a bottleneck since 2 servers are responding back to it which
is why I was curious to see if people used multiple jmeter-clients
instead of jmeter-servers since that would get rid of the multiple hits
to one client. I tried to minimize the bottleneck of the client by
running it in non-gui mode and also using a simple datawriter to write
my results.
Thanks,
Chad
-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: Thursday, August 28, 2008 11:59 AM
To: JMeter Users List
Subject: Re: Question about Scalability of Jmeter-Server
On 28/08/2008, Huesgen, Chad <ch...@litle.com> wrote:
> Hello All,
>
> I am designing some performance tests using jmeter-server and am
getting
> some inconsistent results. My base assumption is that 2 jmeter
servers
> will get better performance numbers then 1 jmeter server or 1 jmeter
> client by spreading the load across.
This is not necessarily true - the servers have to return the data to
the client, so the client can become a bottleneck.
> I am running a very simple ping
> test here basically hitting a Tomcat server with a helloworld.jsp
with a
> 500 milli sleep in it. The setup is I have 2 servers acting as
jmeter
> servers and one server that is hosting Tomcat and the client. The
The server that hosts the application under test should not be used
for anything else - certainly not for running JMeter (client or
server).
> servers only have one hop to each other and all tests run at night so
no
> network traffic should affect the ping. My question is have people
seen
> numbers better and more consistent by spreading jmeter clients out
> across 3 servers and not using jmeter-server? If I saw a Std Dev all
> under 100 and close to each other for each run than it would make me
> feel much better with the results and confident that I can move onto
> designing the real perf tests for the app.
>
> Here are some numbers that I am seeing:
>
>
> Test Case Standard Dev Avg Median Max Min 90ile
> 95ile 99ile
> Jmeter Only 400 Threads 181.0445542 524.6929 503 9507
> 501 504 516 1223
> Jmeter Only 400 Threads 126.5504985 517.8482 503 3508
> 501 504 506 965
> Jmeter Only 400 Threads 213.6036281 534.2436 503 3517
> 501 504 516 1454
>
> One Server 400 Threads 203.1225482 537.1209 504 3577
> 501 558 582 1208
> One Server 400 Threads 211.2885383 545.0142 504 3582
> 501 569 617 1295
> One Server 400 Threads 238.3708453 539.7173 504 9504
> 501 559 582 1279
>
> Two Servers 400 Threads 167.2694959 521.1874 503 5060
> 501 504 505 1422
> Two Servers 400 Threads 175.2300215 523.661 503 3508 501
> 504 506 1310
> Two Servers 400 Threads 187.0677548 521.4626 503 4640
> 501 504 505 1142
>
>
> Jmeter Only 500 Threads 388.3772604 559.2594 503 9509
> 501 507 594 2402
> Jmeter Only 500 Threads 346.868954 549.8935 503 9509
> 501 507 672 1475
> Jmeter Only 500 Threads 340.8531506 554.2277 503 10191
> 501 506 633 1782
>
> One Server 500 Threads 499.4419659 545.5399 504 21513
> 501 560 576 1095
> One Server 500 Threads 230.4885537 546.537 504 3595 501
> 564 578 1752
> One Server 500 Threads 274.5812614 551.7715 505 5331
> 501 564 579 1442
>
> Two Servers 500 Threads 242.6634829 538.9398 503 9503
> 501 504 524 1775
> Two Servers 500 Threads 177.9699861 524.772 503 4398 501
> 504 514 1444
> Two Servers 500 Threads 365.8320039 540.0532 503 22621
> 501 505 525 1587
>
The minimum and median seem very consistent across all the tests.
The variation in the maxima and 9xile columns suggests that there are
some "outlying" data points that are causing problems.
Running the JMeter client on the Tomcat server system will not be
helping here...
For minimum resource usage, use non-GUI mode on multiple independent
hosts and combine the results later.
> Thanks,
>
> Chad
> NOTICE: This message, including all attachments transmitted with it,
is for the use of the addressee only. It may contain proprietary,
confidential and/or legally privileged information belonging to Litle &
Co. No confidentiality or privilege is waived or lost by any
mistransmission. If you are not the intended recipient, you must not,
directly or indirectly, use, disclose, distribute, print or copy any
part of this message. If you believe you have received this message in
error, please delete it and all copies of it from your system and notify
the sender immediately by reply e-mail. Thank you.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Question about Scalability of Jmeter-Server
Posted by sebb <se...@gmail.com>.
On 28/08/2008, Huesgen, Chad <ch...@litle.com> wrote:
> Hello All,
>
> I am designing some performance tests using jmeter-server and am getting
> some inconsistent results. My base assumption is that 2 jmeter servers
> will get better performance numbers then 1 jmeter server or 1 jmeter
> client by spreading the load across.
This is not necessarily true - the servers have to return the data to
the client, so the client can become a bottleneck.
> I am running a very simple ping
> test here basically hitting a Tomcat server with a helloworld.jsp with a
> 500 milli sleep in it. The setup is I have 2 servers acting as jmeter
> servers and one server that is hosting Tomcat and the client. The
The server that hosts the application under test should not be used
for anything else - certainly not for running JMeter (client or
server).
> servers only have one hop to each other and all tests run at night so no
> network traffic should affect the ping. My question is have people seen
> numbers better and more consistent by spreading jmeter clients out
> across 3 servers and not using jmeter-server? If I saw a Std Dev all
> under 100 and close to each other for each run than it would make me
> feel much better with the results and confident that I can move onto
> designing the real perf tests for the app.
>
> Here are some numbers that I am seeing:
>
>
> Test Case Standard Dev Avg Median Max Min 90ile
> 95ile 99ile
> Jmeter Only 400 Threads 181.0445542 524.6929 503 9507
> 501 504 516 1223
> Jmeter Only 400 Threads 126.5504985 517.8482 503 3508
> 501 504 506 965
> Jmeter Only 400 Threads 213.6036281 534.2436 503 3517
> 501 504 516 1454
>
> One Server 400 Threads 203.1225482 537.1209 504 3577
> 501 558 582 1208
> One Server 400 Threads 211.2885383 545.0142 504 3582
> 501 569 617 1295
> One Server 400 Threads 238.3708453 539.7173 504 9504
> 501 559 582 1279
>
> Two Servers 400 Threads 167.2694959 521.1874 503 5060
> 501 504 505 1422
> Two Servers 400 Threads 175.2300215 523.661 503 3508 501
> 504 506 1310
> Two Servers 400 Threads 187.0677548 521.4626 503 4640
> 501 504 505 1142
>
>
> Jmeter Only 500 Threads 388.3772604 559.2594 503 9509
> 501 507 594 2402
> Jmeter Only 500 Threads 346.868954 549.8935 503 9509
> 501 507 672 1475
> Jmeter Only 500 Threads 340.8531506 554.2277 503 10191
> 501 506 633 1782
>
> One Server 500 Threads 499.4419659 545.5399 504 21513
> 501 560 576 1095
> One Server 500 Threads 230.4885537 546.537 504 3595 501
> 564 578 1752
> One Server 500 Threads 274.5812614 551.7715 505 5331
> 501 564 579 1442
>
> Two Servers 500 Threads 242.6634829 538.9398 503 9503
> 501 504 524 1775
> Two Servers 500 Threads 177.9699861 524.772 503 4398 501
> 504 514 1444
> Two Servers 500 Threads 365.8320039 540.0532 503 22621
> 501 505 525 1587
>
The minimum and median seem very consistent across all the tests.
The variation in the maxima and 9xile columns suggests that there are
some "outlying" data points that are causing problems.
Running the JMeter client on the Tomcat server system will not be
helping here...
For minimum resource usage, use non-GUI mode on multiple independent
hosts and combine the results later.
> Thanks,
>
> Chad
> NOTICE: This message, including all attachments transmitted with it, is for the use of the addressee only. It may contain proprietary, confidential and/or legally privileged information belonging to Litle & Co. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient, you must not, directly or indirectly, use, disclose, distribute, print or copy any part of this message. If you believe you have received this message in error, please delete it and all copies of it from your system and notify the sender immediately by reply e-mail. Thank you.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org