jmeter - thread count limit

[Shilpa Kulkarni <sy...@payasonline.com>]

Hi

I have a question about thread count limit in jmeter. Some of the -
slightly older documentation on the internet - lists a number between
200-300 for number of threads. Some docs mention there is no hard & fast
rule, it depends on system hardware/software config & how soon the server
responds for APIs under test.

I have built a basic API test plan for a mobile/server type of application
& tested for 2 users in csv file & thread count = 10. We now need to put
more load on it & I was checking out for any guide lines here. What are my
options? A + B or just one of them? Are there any other options?
A. Increase number of users in csv file
B. Increase thread count

Thanks
SK

Re: jmeter - thread count limit

[Shilpa Kulkarni <sy...@payasonline.com>]

Thank you for your quick responses. We have just gotten started on the load
testing effort & relatively new to jmeter/load test area. Just from my
handful number of days on this mailing list & seeing the community support
is very encouraging.

Thanks again!

SK


On Wed, Aug 13, 2014 at 3:22 AM, Adrian Speteanu <as...@gmail.com>
wrote:

> Hi,
>
> +1 to the previous responses, in short you're going to have to experiment
> with things and decide what works best for you.
>
> I can only share my experience with rest APIs, though my example is on the
> extreme. In the case I will present, the roundtrip of a single request
> takes 2-10-20ms and the added size of the request + it's response is
> smaller than 1kb. But the app needs to handle many concurrent active users.
>
> To generate and handle the kind of load that was required for such a test,
> we used actual physical machines (2x quad core, with 24Gb RAM servers) or
> EC2 instances (the extra large memory type of instance). And we were able
> to generate a maximum throughput of 30 thousands requests per second, on a
> single test client (using the physical machines). The test was able to
> sustain this value for long periods of time (but it required supervision).
> Assertions and logging done by the test script was reduced to a bare
> minimum, enough to make sure that the script is actually executing what we
> think it does.
>
> After experimentations, the test script was configured to run smoothly in a
> 2Gb Max Heap memory instance of jmeter. It spawned 1000-3000 threads and it
> ran smoothly (was able to generate constant throughput, when the test
> target was stable, for long periods of time, 12-48h and the amount of
> errors that were related to the test client was negligible). To generate
> 30k requests, 10-12 instances of jmeter were used and the machine CPU usage
> was 70-90%, but there were many tweaks on the OS side we had to make,
> mostly related to the network configuration (after we ran into various
> issues). The idea of a 2Gb instance was to be able to scale horizontally
> though. In this case, scaling vertically meant using more jmeter instances
> and this kind of setup proved more stable and easier to maintain.
>
> So that was the long version of: jmeter can handle as many threads as the
> JVM can on that particular machine, which is quite a lot.
>
> But with more complex applications, things will get complicated. The bigger
> the response, the more processing you require within jmeter, the more
> resources it will require and the number of threads you'll be able to spawn
> in a stable setup will decrease dramatically. For something like a website,
> you can overwhelm 1 frontend instance of the target server with 50-100
> threads (but they will consume a lot of CPU on your test client).
>
> --Adrian
>
>
> On Tue, Aug 12, 2014 at 6:24 PM, Deepak Shetty <sh...@gmail.com> wrote:
>
> > Hi
> > >"docs mention there is no hard & fast rule, it depends on system
> > hardware/software config & how soon the server responds for APIs under
> > test."
> > This is the technically correct answer
> >
> > >" lists a number between 200-300 for number of threads"
> > This is most likely a rule of thumb or a starting point (e.g. I will
> rarely
> > exceed 40 threads on my client PC)
> >
> > >We now need to put more load on it
> > It depends on how you define "load" - *Usually* you mean the concurrent
> > usage that the server sees in which case option B is the correct
> one.(It's
> > possible you might be running a long running test , just to see whether
> > there are memory leaks or something like that and you are using "load" in
> > that context in which case you probably only need option A. - happens ,
> but
> > rare)
> > However lets say you want to test out 10 users accessing your system at
> the
> > same time, for an hour , then you might want to use different users
> (rather
> > than using the same user or  10 users which means you might get better
> than
> > expected results due to server caching user details or something like
> that)
> > - in which case you do also need Option A to make Option B realistic.
> > However you cannot just increase the number of threads without limit ,
> > because at some point of time , either the JMeter machine or the network
> > might start slowing down your tests which would not happen in a real life
> > scenario - in which case you now need to add more JMeter machines (either
> > distributed or standalone) - What number of threads your system supports
> is
> > dependent on multiple factors and there is no way out of testing and
> > validating your results - This has probably been discussed numerous times
> > in the archives of these mailing lists.
> >
> >
> >
> > regards
> > deepak
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 11, 2014 at 6:06 PM, Shilpa Kulkarni <sy...@payasonline.com>
> > wrote:
> >
> > > Hi
> > >
> > > I have a question about thread count limit in jmeter. Some of the -
> > > slightly older documentation on the internet - lists a number between
> > > 200-300 for number of threads. Some docs mention there is no hard &
> fast
> > > rule, it depends on system hardware/software config & how soon the
> server
> > > responds for APIs under test.
> > >
> > > I have built a basic API test plan for a mobile/server type of
> > application
> > > & tested for 2 users in csv file & thread count = 10. We now need to
> put
> > > more load on it & I was checking out for any guide lines here. What are
> > my
> > > options? A + B or just one of them? Are there any other options?
> > > A. Increase number of users in csv file
> > > B. Increase thread count
> > >
> > > Thanks
> > > SK
> > >
> >
>

Re: jmeter - thread count limit

[Adrian Speteanu <as...@gmail.com>]

Hi,

+1 to the previous responses, in short you're going to have to experiment
with things and decide what works best for you.

I can only share my experience with rest APIs, though my example is on the
extreme. In the case I will present, the roundtrip of a single request
takes 2-10-20ms and the added size of the request + it's response is
smaller than 1kb. But the app needs to handle many concurrent active users.

To generate and handle the kind of load that was required for such a test,
we used actual physical machines (2x quad core, with 24Gb RAM servers) or
EC2 instances (the extra large memory type of instance). And we were able
to generate a maximum throughput of 30 thousands requests per second, on a
single test client (using the physical machines). The test was able to
sustain this value for long periods of time (but it required supervision).
Assertions and logging done by the test script was reduced to a bare
minimum, enough to make sure that the script is actually executing what we
think it does.

After experimentations, the test script was configured to run smoothly in a
2Gb Max Heap memory instance of jmeter. It spawned 1000-3000 threads and it
ran smoothly (was able to generate constant throughput, when the test
target was stable, for long periods of time, 12-48h and the amount of
errors that were related to the test client was negligible). To generate
30k requests, 10-12 instances of jmeter were used and the machine CPU usage
was 70-90%, but there were many tweaks on the OS side we had to make,
mostly related to the network configuration (after we ran into various
issues). The idea of a 2Gb instance was to be able to scale horizontally
though. In this case, scaling vertically meant using more jmeter instances
and this kind of setup proved more stable and easier to maintain.

So that was the long version of: jmeter can handle as many threads as the
JVM can on that particular machine, which is quite a lot.

But with more complex applications, things will get complicated. The bigger
the response, the more processing you require within jmeter, the more
resources it will require and the number of threads you'll be able to spawn
in a stable setup will decrease dramatically. For something like a website,
you can overwhelm 1 frontend instance of the target server with 50-100
threads (but they will consume a lot of CPU on your test client).

--Adrian


On Tue, Aug 12, 2014 at 6:24 PM, Deepak Shetty <sh...@gmail.com> wrote:

> Hi
> >"docs mention there is no hard & fast rule, it depends on system
> hardware/software config & how soon the server responds for APIs under
> test."
> This is the technically correct answer
>
> >" lists a number between 200-300 for number of threads"
> This is most likely a rule of thumb or a starting point (e.g. I will rarely
> exceed 40 threads on my client PC)
>
> >We now need to put more load on it
> It depends on how you define "load" - *Usually* you mean the concurrent
> usage that the server sees in which case option B is the correct one.(It's
> possible you might be running a long running test , just to see whether
> there are memory leaks or something like that and you are using "load" in
> that context in which case you probably only need option A. - happens , but
> rare)
> However lets say you want to test out 10 users accessing your system at the
> same time, for an hour , then you might want to use different users (rather
> than using the same user or  10 users which means you might get better than
> expected results due to server caching user details or something like that)
> - in which case you do also need Option A to make Option B realistic.
> However you cannot just increase the number of threads without limit ,
> because at some point of time , either the JMeter machine or the network
> might start slowing down your tests which would not happen in a real life
> scenario - in which case you now need to add more JMeter machines (either
> distributed or standalone) - What number of threads your system supports is
> dependent on multiple factors and there is no way out of testing and
> validating your results - This has probably been discussed numerous times
> in the archives of these mailing lists.
>
>
>
> regards
> deepak
>
>
>
>
>
>
>
>
> On Mon, Aug 11, 2014 at 6:06 PM, Shilpa Kulkarni <sy...@payasonline.com>
> wrote:
>
> > Hi
> >
> > I have a question about thread count limit in jmeter. Some of the -
> > slightly older documentation on the internet - lists a number between
> > 200-300 for number of threads. Some docs mention there is no hard & fast
> > rule, it depends on system hardware/software config & how soon the server
> > responds for APIs under test.
> >
> > I have built a basic API test plan for a mobile/server type of
> application
> > & tested for 2 users in csv file & thread count = 10. We now need to put
> > more load on it & I was checking out for any guide lines here. What are
> my
> > options? A + B or just one of them? Are there any other options?
> > A. Increase number of users in csv file
> > B. Increase thread count
> >
> > Thanks
> > SK
> >
>

Re: jmeter - thread count limit

[Deepak Shetty <sh...@gmail.com>]

Hi
>"docs mention there is no hard & fast rule, it depends on system
hardware/software config & how soon the server responds for APIs under
test."
This is the technically correct answer

>" lists a number between 200-300 for number of threads"
This is most likely a rule of thumb or a starting point (e.g. I will rarely
exceed 40 threads on my client PC)

>We now need to put more load on it
It depends on how you define "load" - *Usually* you mean the concurrent
usage that the server sees in which case option B is the correct one.(It's
possible you might be running a long running test , just to see whether
there are memory leaks or something like that and you are using "load" in
that context in which case you probably only need option A. - happens , but
rare)
However lets say you want to test out 10 users accessing your system at the
same time, for an hour , then you might want to use different users (rather
than using the same user or  10 users which means you might get better than
expected results due to server caching user details or something like that)
- in which case you do also need Option A to make Option B realistic.
However you cannot just increase the number of threads without limit ,
because at some point of time , either the JMeter machine or the network
might start slowing down your tests which would not happen in a real life
scenario - in which case you now need to add more JMeter machines (either
distributed or standalone) - What number of threads your system supports is
dependent on multiple factors and there is no way out of testing and
validating your results - This has probably been discussed numerous times
in the archives of these mailing lists.



regards
deepak








On Mon, Aug 11, 2014 at 6:06 PM, Shilpa Kulkarni <sy...@payasonline.com>
wrote:

> Hi
>
> I have a question about thread count limit in jmeter. Some of the -
> slightly older documentation on the internet - lists a number between
> 200-300 for number of threads. Some docs mention there is no hard & fast
> rule, it depends on system hardware/software config & how soon the server
> responds for APIs under test.
>
> I have built a basic API test plan for a mobile/server type of application
> & tested for 2 users in csv file & thread count = 10. We now need to put
> more load on it & I was checking out for any guide lines here. What are my
> options? A + B or just one of them? Are there any other options?
> A. Increase number of users in csv file
> B. Increase thread count
>
> Thanks
> SK
>

Re: jmeter - thread count limit

[Niraj <ni...@gmail.com>]

Why don't you try out both increasing A+B options. And then probably you
can reach to conclusion
On Aug 12, 2014 6:36 AM, "Shilpa Kulkarni" <sy...@payasonline.com> wrote:

> Hi
>
> I have a question about thread count limit in jmeter. Some of the -
> slightly older documentation on the internet - lists a number between
> 200-300 for number of threads. Some docs mention there is no hard & fast
> rule, it depends on system hardware/software config & how soon the server
> responds for APIs under test.
>
> I have built a basic API test plan for a mobile/server type of application
> & tested for 2 users in csv file & thread count = 10. We now need to put
> more load on it & I was checking out for any guide lines here. What are my
> options? A + B or just one of them? Are there any other options?
> A. Increase number of users in csv file
> B. Increase thread count
>
> Thanks
> SK
>