You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by David Luu <ma...@gmail.com> on 2012/10/16 03:26:26 UTC
Some assumptions on JMeter limitations. Your thoughts?
Hello,
I was asking for feedback on our usage of JMeter within my organization and
got some responses with some disadvantages/limitations of JMeter. Based on
the responses, most of it to me seems like some assumptions made w/o
knowing/researching the full capabilities of JMeter. It took me some time
to explore and find out JMeter's feature set.
But in any case thought I'd ask the community for input regarding these
(assumed?) disadvantages, extracted snippet of feedback:
"I don't expect us to use JMeter long-term because it is pretty difficult
to:
* extract and re-use components, e.g. sign-in
* verify or review the test or your changes to it are doing what you think
it is doing; you're stuck with either understanding the test through the UI
or reading jmx directly
Perhaps you can share some insights in how you have addressed the above
points.
I think one way to address these issues is to express our performance/load
tests in a real programming environment. This would give us the normal
power we expect to manage our test software. There are a few systems that
allow for this, e.g. gatling [1], grinder [2], iago [3]
[1] http://gatling-tool.org/
[2] http://grinder.sourceforge.net/
[3] http://twitter.github.com/iago/
...
Also, having a programming language support gives you a better control on
the load testing scripts, but Jmeter doesn’t have that."
The re-use issue seems interesting to me. How best to do that in JMeter?
For me all I could think of is building a standard test template with
common components and settings and everyone works off a copy of that
customizing as needed. But is there a better way to reuse-share a component
like HTTP header manager, or User Parameters or User Defined Variables, or
Regular Expression Extractor, etc. with the customized variables, values,
patterns w/o having to copy paste from one test plan to another?
On the understanding/reading JMeter test plan, my only thought to that was
good test plan design in terms of renaming components in test plan,
labelling them with description of what that component is doing in the test
and adding in info in the comment fields for components that have them. And
that would apply to both reviewing test plan in GUI and from parsing JMX
file. Any thoughts beyond that?
For the programmatic functionality area, I know we can get that by using
JMeter custom plugins, or import Java libraries for use with
JUnit/BSF/Beanshell pre/post processors, assertions, and samplers. Build
out the needed functionality in code & call it from JMeter, using JMeter
for threading, concurrency, load management, and possibly reporting/logging
too. But any thoughts beyond that? I'm assuming my colleagues were looking
more for a full featured IDE type tool or language binding based framework
like Selenium 2/WebDriver where you write in code using "JMeter" APIs to do
the load testing.
Regards,
David
RE: Some assumptions on JMeter limitations. Your thoughts?
Posted by "BOLB (Bohdan L Bodnar)" <BO...@panduit.com>.
Here are my $0.02:
- lack of dedicated technical support (unlike using commercial tools)
- unable to integrate with other (commercial) tools (e.g., IBM's "Jazz")
Otherwise, jmeter appears to me as a good tool (and it's free). For me, the first item is the thing that bugs me the most. I'm not a proficient user of jmeter and end up spending a lot of time looking at postings about how to do things; a compilation of examples, ranging from trivial to very elaborate, would be nice. For example, an idiot-proof example on how to pass parameters between threads would be nice.
I don't bother with jmeter's graphing - I dump the output to a .csv file and then read it using (commercial) data mining tools to do analysis much more sophisticated than jmeter is capable of doing (e.g., paired t-tests, cross-correlations, distribution fits, etc.). Hence, I cannot comment on how useful the graphing is to me.
Overall, it's a good tool.
Hope this helps,
Bo
-----Original Message-----
From: David Luu [mailto:mangaroo@gmail.com]
Sent: Monday, October 15, 2012 8:26 PM
To: JMeter Users List
Subject: Some assumptions on JMeter limitations. Your thoughts?
Hello,
I was asking for feedback on our usage of JMeter within my organization and got some responses with some disadvantages/limitations of JMeter. Based on the responses, most of it to me seems like some assumptions made w/o knowing/researching the full capabilities of JMeter. It took me some time to explore and find out JMeter's feature set.
But in any case thought I'd ask the community for input regarding these
(assumed?) disadvantages, extracted snippet of feedback:
"I don't expect us to use JMeter long-term because it is pretty difficult
to:
* extract and re-use components, e.g. sign-in
* verify or review the test or your changes to it are doing what you think it is doing; you're stuck with either understanding the test through the UI or reading jmx directly
Perhaps you can share some insights in how you have addressed the above points.
I think one way to address these issues is to express our performance/load tests in a real programming environment. This would give us the normal power we expect to manage our test software. There are a few systems that allow for this, e.g. gatling [1], grinder [2], iago [3]
[1] http://gatling-tool.org/
[2] http://grinder.sourceforge.net/
[3] http://twitter.github.com/iago/
...
Also, having a programming language support gives you a better control on the load testing scripts, but Jmeter doesn't have that."
The re-use issue seems interesting to me. How best to do that in JMeter?
For me all I could think of is building a standard test template with common components and settings and everyone works off a copy of that customizing as needed. But is there a better way to reuse-share a component like HTTP header manager, or User Parameters or User Defined Variables, or Regular Expression Extractor, etc. with the customized variables, values, patterns w/o having to copy paste from one test plan to another?
On the understanding/reading JMeter test plan, my only thought to that was good test plan design in terms of renaming components in test plan, labelling them with description of what that component is doing in the test and adding in info in the comment fields for components that have them. And that would apply to both reviewing test plan in GUI and from parsing JMX file. Any thoughts beyond that?
For the programmatic functionality area, I know we can get that by using JMeter custom plugins, or import Java libraries for use with JUnit/BSF/Beanshell pre/post processors, assertions, and samplers. Build out the needed functionality in code & call it from JMeter, using JMeter for threading, concurrency, load management, and possibly reporting/logging too. But any thoughts beyond that? I'm assuming my colleagues were looking more for a full featured IDE type tool or language binding based framework like Selenium 2/WebDriver where you write in code using "JMeter" APIs to do the load testing.
Regards,
David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Some assumptions on JMeter limitations. Your thoughts?
Posted by Adrian Speteanu <as...@gmail.com>.
Hi Oliver,
I use it too, but for big data sets, I still use external tools.
Cheers,
--Adrtian S
On Thu, Oct 18, 2012 at 12:18 PM, Oliver Erlewein <ol...@erlewein.net>wrote:
> Hi
>
> Have you looked at the JMeter Plug-ins?
> http://code.google.com/p/jmeter-plugins/
> That might help your graphing issues. I also use gnuplot to do most of my
> graphing.
>
> Cheers Oliver
>
> On 18 October 2012 22:07, Adrian Speteanu <as...@gmail.com> wrote:
>
> > Hi,
> >
> > Disadvantages:
> > - native graphing, plotting capabilities are a real problem to most
> > beginners; of course you can workaround it by using other data processing
> > tools (I've used a lot of solutions Excel, jmeter plugins, external
> > scripting, app-under-test monitoring tools - which are the best approach
> > IMO -, exporting to MySQL, and recently company dedicated data processing
> > tools); its costly to have good graphing tools, I could also live without
> > it just fine, if I would lack the time and manpower to set this up - in
> > this case I would just use statistics, default and plugin listeners.
> > - its difficult to maintain large functional test scripts; again, this
> > is not a big deal for an experienced tester: i.e. smaller jmeter scripts
> > integrated in jenkins work just fine. Reference is possible, but not easy
> > to cope with and digest. This means that you need to use non-jmeter ways
> to
> > manage your scripts that form a larger test plan. Personally, I
> appreciate
> > this freedom.
> > - its difficult for people used with other tools and some devs [I've
> > worked with] to get used to it's UI, even when they see the advantages of
> > using it. I really don't see how the interface should be changed, it
> works
> > for me, but I've had to explain every now and then how the tree-like
> > structure affects execution order, how the threads work, how timers
> affect
> > their scope. Most of the times I refer to the documentation myself when
> > adding an element that I haven't worked recently with. And the
> > documentation is clear and short - which makes this quite ok.
> >
> > Comment on previous messages:
> > Extending JMeter - its functionality or your test's functionality - is
> > so extremely EASY. Its the best thing about it. Whatever you don't like
> or
> > can't do natively, you just wright code for it. Apart the fact that it
> can
> > be used for so many different web apps, this is probably its big
> technical
> > advantage. Most tech-geeks I know prefer it for performance tests.
> >
> > Regards,
> > --Adrian S
> >
> > On Tue, Oct 16, 2012 at 5:35 PM, Anthony Johnson <an...@gmail.com>
> wrote:
> >
> > > My thoughts below:-)
> > >
> > > On Mon, Oct 15, 2012 at 9:26 PM, David Luu <ma...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > I was asking for feedback on our usage of JMeter within my
> organization
> > > and
> > > > got some responses with some disadvantages/limitations of JMeter.
> Based
> > > on
> > > > the responses, most of it to me seems like some assumptions made w/o
> > > > knowing/researching the full capabilities of JMeter. It took me some
> > time
> > > > to explore and find out JMeter's feature set.
> > > >
> > > > But in any case thought I'd ask the community for input regarding
> these
> > > > (assumed?) disadvantages, extracted snippet of feedback:
> > > >
> > > > "I don't expect us to use JMeter long-term because it is pretty
> > difficult
> > > > to:
> > > >
> > > > * extract and re-use components, e.g. sign-in
> > >
> > > You can create mini-test plans that have a Test Fragment as the base
> > > element and then load those via Include Controllers in the main Test
> > > Plan. This allows you to create all sorts of re-usable components.
> > >
> > > > * verify or review the test or your changes to it are doing what you
> > > think
> > > > it is doing; you're stuck with either understanding the test through
> > the
> > > UI
> > > > or reading jmx directly
> > >
> > > Well-Written plans that use the Descriptions and name the components
> > > properly read very well. This is just discipline in using the tool
> > > IMO.
> > >
> > > >
> > > > Perhaps you can share some insights in how you have addressed the
> above
> > > > points.
> > > >
> > > > I think one way to address these issues is to express our
> > > performance/load
> > > > tests in a real programming environment. This would give us the
> normal
> > > > power we expect to manage our test software. There are a few systems
> > > that
> > > > allow for this, e.g. gatling [1], grinder [2], iago [3]
> > > >
> > > > [1] http://gatling-tool.org/
> > > > [2] http://grinder.sourceforge.net/
> > > > [3] http://twitter.github.com/iago/
> > > > ...
> > > > Also, having a programming language support gives you a better
> control
> > on
> > > > the load testing scripts, but Jmeter doesn’t have that."
> > >
> > > You can use some Scripting in JMeter, but I purposely chose JMeter to
> > > avoid having to write code. The code barrier sets a baseline for who
> > > can create and understand Performance tests. In my environment, QA
> > > Engineers with no programming experience are able to create, execute
> > > and debug performance tests. Abstracting all the programming with a
> > > usable UI is JMeter's best feature IMO.
> > >
> > > >
> > > > The re-use issue seems interesting to me. How best to do that in
> > JMeter?
> > > > For me all I could think of is building a standard test template with
> > > > common components and settings and everyone works off a copy of that
> > > > customizing as needed. But is there a better way to reuse-share a
> > > component
> > > > like HTTP header manager, or User Parameters or User Defined
> Variables,
> > > or
> > > > Regular Expression Extractor, etc. with the customized variables,
> > values,
> > > > patterns w/o having to copy paste from one test plan to another?
> > >
> > > Include Controller. I think the project could use some better
> > > documentation around this and also ship with some examples.
> > >
> > > >
> > > > On the understanding/reading JMeter test plan, my only thought to
> that
> > > was
> > > > good test plan design in terms of renaming components in test plan,
> > > > labelling them with description of what that component is doing in
> the
> > > test
> > > > and adding in info in the comment fields for components that have
> them.
> > > And
> > > > that would apply to both reviewing test plan in GUI and from parsing
> > JMX
> > > > file. Any thoughts beyond that?
> > >
> > > This is how I do it.
> > >
> > > >
> > > > For the programmatic functionality area, I know we can get that by
> > using
> > > > JMeter custom plugins, or import Java libraries for use with
> > > > JUnit/BSF/Beanshell pre/post processors, assertions, and samplers.
> > Build
> > > > out the needed functionality in code & call it from JMeter, using
> > JMeter
> > > > for threading, concurrency, load management, and possibly
> > > reporting/logging
> > > > too. But any thoughts beyond that? I'm assuming my colleagues were
> > > looking
> > > > more for a full featured IDE type tool or language binding based
> > > framework
> > > > like Selenium 2/WebDriver where you write in code using "JMeter" APIs
> > to
> > > do
> > > > the load testing.
> > > >
> > > > Regards,
> > > > David
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > For additional commands, e-mail: user-help@jmeter.apache.org
> > >
> > >
> >
>
Re: Some assumptions on JMeter limitations. Your thoughts?
Posted by Oliver Erlewein <ol...@erlewein.net>.
Hi
Have you looked at the JMeter Plug-ins?
http://code.google.com/p/jmeter-plugins/
That might help your graphing issues. I also use gnuplot to do most of my
graphing.
Cheers Oliver
On 18 October 2012 22:07, Adrian Speteanu <as...@gmail.com> wrote:
> Hi,
>
> Disadvantages:
> - native graphing, plotting capabilities are a real problem to most
> beginners; of course you can workaround it by using other data processing
> tools (I've used a lot of solutions Excel, jmeter plugins, external
> scripting, app-under-test monitoring tools - which are the best approach
> IMO -, exporting to MySQL, and recently company dedicated data processing
> tools); its costly to have good graphing tools, I could also live without
> it just fine, if I would lack the time and manpower to set this up - in
> this case I would just use statistics, default and plugin listeners.
> - its difficult to maintain large functional test scripts; again, this
> is not a big deal for an experienced tester: i.e. smaller jmeter scripts
> integrated in jenkins work just fine. Reference is possible, but not easy
> to cope with and digest. This means that you need to use non-jmeter ways to
> manage your scripts that form a larger test plan. Personally, I appreciate
> this freedom.
> - its difficult for people used with other tools and some devs [I've
> worked with] to get used to it's UI, even when they see the advantages of
> using it. I really don't see how the interface should be changed, it works
> for me, but I've had to explain every now and then how the tree-like
> structure affects execution order, how the threads work, how timers affect
> their scope. Most of the times I refer to the documentation myself when
> adding an element that I haven't worked recently with. And the
> documentation is clear and short - which makes this quite ok.
>
> Comment on previous messages:
> Extending JMeter - its functionality or your test's functionality - is
> so extremely EASY. Its the best thing about it. Whatever you don't like or
> can't do natively, you just wright code for it. Apart the fact that it can
> be used for so many different web apps, this is probably its big technical
> advantage. Most tech-geeks I know prefer it for performance tests.
>
> Regards,
> --Adrian S
>
> On Tue, Oct 16, 2012 at 5:35 PM, Anthony Johnson <an...@gmail.com> wrote:
>
> > My thoughts below:-)
> >
> > On Mon, Oct 15, 2012 at 9:26 PM, David Luu <ma...@gmail.com> wrote:
> > > Hello,
> > >
> > > I was asking for feedback on our usage of JMeter within my organization
> > and
> > > got some responses with some disadvantages/limitations of JMeter. Based
> > on
> > > the responses, most of it to me seems like some assumptions made w/o
> > > knowing/researching the full capabilities of JMeter. It took me some
> time
> > > to explore and find out JMeter's feature set.
> > >
> > > But in any case thought I'd ask the community for input regarding these
> > > (assumed?) disadvantages, extracted snippet of feedback:
> > >
> > > "I don't expect us to use JMeter long-term because it is pretty
> difficult
> > > to:
> > >
> > > * extract and re-use components, e.g. sign-in
> >
> > You can create mini-test plans that have a Test Fragment as the base
> > element and then load those via Include Controllers in the main Test
> > Plan. This allows you to create all sorts of re-usable components.
> >
> > > * verify or review the test or your changes to it are doing what you
> > think
> > > it is doing; you're stuck with either understanding the test through
> the
> > UI
> > > or reading jmx directly
> >
> > Well-Written plans that use the Descriptions and name the components
> > properly read very well. This is just discipline in using the tool
> > IMO.
> >
> > >
> > > Perhaps you can share some insights in how you have addressed the above
> > > points.
> > >
> > > I think one way to address these issues is to express our
> > performance/load
> > > tests in a real programming environment. This would give us the normal
> > > power we expect to manage our test software. There are a few systems
> > that
> > > allow for this, e.g. gatling [1], grinder [2], iago [3]
> > >
> > > [1] http://gatling-tool.org/
> > > [2] http://grinder.sourceforge.net/
> > > [3] http://twitter.github.com/iago/
> > > ...
> > > Also, having a programming language support gives you a better control
> on
> > > the load testing scripts, but Jmeter doesn’t have that."
> >
> > You can use some Scripting in JMeter, but I purposely chose JMeter to
> > avoid having to write code. The code barrier sets a baseline for who
> > can create and understand Performance tests. In my environment, QA
> > Engineers with no programming experience are able to create, execute
> > and debug performance tests. Abstracting all the programming with a
> > usable UI is JMeter's best feature IMO.
> >
> > >
> > > The re-use issue seems interesting to me. How best to do that in
> JMeter?
> > > For me all I could think of is building a standard test template with
> > > common components and settings and everyone works off a copy of that
> > > customizing as needed. But is there a better way to reuse-share a
> > component
> > > like HTTP header manager, or User Parameters or User Defined Variables,
> > or
> > > Regular Expression Extractor, etc. with the customized variables,
> values,
> > > patterns w/o having to copy paste from one test plan to another?
> >
> > Include Controller. I think the project could use some better
> > documentation around this and also ship with some examples.
> >
> > >
> > > On the understanding/reading JMeter test plan, my only thought to that
> > was
> > > good test plan design in terms of renaming components in test plan,
> > > labelling them with description of what that component is doing in the
> > test
> > > and adding in info in the comment fields for components that have them.
> > And
> > > that would apply to both reviewing test plan in GUI and from parsing
> JMX
> > > file. Any thoughts beyond that?
> >
> > This is how I do it.
> >
> > >
> > > For the programmatic functionality area, I know we can get that by
> using
> > > JMeter custom plugins, or import Java libraries for use with
> > > JUnit/BSF/Beanshell pre/post processors, assertions, and samplers.
> Build
> > > out the needed functionality in code & call it from JMeter, using
> JMeter
> > > for threading, concurrency, load management, and possibly
> > reporting/logging
> > > too. But any thoughts beyond that? I'm assuming my colleagues were
> > looking
> > > more for a full featured IDE type tool or language binding based
> > framework
> > > like Selenium 2/WebDriver where you write in code using "JMeter" APIs
> to
> > do
> > > the load testing.
> > >
> > > Regards,
> > > David
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
> >
>
Re: Some assumptions on JMeter limitations. Your thoughts?
Posted by Adrian Speteanu <as...@gmail.com>.
Hi,
Disadvantages:
- native graphing, plotting capabilities are a real problem to most
beginners; of course you can workaround it by using other data processing
tools (I've used a lot of solutions Excel, jmeter plugins, external
scripting, app-under-test monitoring tools - which are the best approach
IMO -, exporting to MySQL, and recently company dedicated data processing
tools); its costly to have good graphing tools, I could also live without
it just fine, if I would lack the time and manpower to set this up - in
this case I would just use statistics, default and plugin listeners.
- its difficult to maintain large functional test scripts; again, this
is not a big deal for an experienced tester: i.e. smaller jmeter scripts
integrated in jenkins work just fine. Reference is possible, but not easy
to cope with and digest. This means that you need to use non-jmeter ways to
manage your scripts that form a larger test plan. Personally, I appreciate
this freedom.
- its difficult for people used with other tools and some devs [I've
worked with] to get used to it's UI, even when they see the advantages of
using it. I really don't see how the interface should be changed, it works
for me, but I've had to explain every now and then how the tree-like
structure affects execution order, how the threads work, how timers affect
their scope. Most of the times I refer to the documentation myself when
adding an element that I haven't worked recently with. And the
documentation is clear and short - which makes this quite ok.
Comment on previous messages:
Extending JMeter - its functionality or your test's functionality - is
so extremely EASY. Its the best thing about it. Whatever you don't like or
can't do natively, you just wright code for it. Apart the fact that it can
be used for so many different web apps, this is probably its big technical
advantage. Most tech-geeks I know prefer it for performance tests.
Regards,
--Adrian S
On Tue, Oct 16, 2012 at 5:35 PM, Anthony Johnson <an...@gmail.com> wrote:
> My thoughts below:-)
>
> On Mon, Oct 15, 2012 at 9:26 PM, David Luu <ma...@gmail.com> wrote:
> > Hello,
> >
> > I was asking for feedback on our usage of JMeter within my organization
> and
> > got some responses with some disadvantages/limitations of JMeter. Based
> on
> > the responses, most of it to me seems like some assumptions made w/o
> > knowing/researching the full capabilities of JMeter. It took me some time
> > to explore and find out JMeter's feature set.
> >
> > But in any case thought I'd ask the community for input regarding these
> > (assumed?) disadvantages, extracted snippet of feedback:
> >
> > "I don't expect us to use JMeter long-term because it is pretty difficult
> > to:
> >
> > * extract and re-use components, e.g. sign-in
>
> You can create mini-test plans that have a Test Fragment as the base
> element and then load those via Include Controllers in the main Test
> Plan. This allows you to create all sorts of re-usable components.
>
> > * verify or review the test or your changes to it are doing what you
> think
> > it is doing; you're stuck with either understanding the test through the
> UI
> > or reading jmx directly
>
> Well-Written plans that use the Descriptions and name the components
> properly read very well. This is just discipline in using the tool
> IMO.
>
> >
> > Perhaps you can share some insights in how you have addressed the above
> > points.
> >
> > I think one way to address these issues is to express our
> performance/load
> > tests in a real programming environment. This would give us the normal
> > power we expect to manage our test software. There are a few systems
> that
> > allow for this, e.g. gatling [1], grinder [2], iago [3]
> >
> > [1] http://gatling-tool.org/
> > [2] http://grinder.sourceforge.net/
> > [3] http://twitter.github.com/iago/
> > ...
> > Also, having a programming language support gives you a better control on
> > the load testing scripts, but Jmeter doesn’t have that."
>
> You can use some Scripting in JMeter, but I purposely chose JMeter to
> avoid having to write code. The code barrier sets a baseline for who
> can create and understand Performance tests. In my environment, QA
> Engineers with no programming experience are able to create, execute
> and debug performance tests. Abstracting all the programming with a
> usable UI is JMeter's best feature IMO.
>
> >
> > The re-use issue seems interesting to me. How best to do that in JMeter?
> > For me all I could think of is building a standard test template with
> > common components and settings and everyone works off a copy of that
> > customizing as needed. But is there a better way to reuse-share a
> component
> > like HTTP header manager, or User Parameters or User Defined Variables,
> or
> > Regular Expression Extractor, etc. with the customized variables, values,
> > patterns w/o having to copy paste from one test plan to another?
>
> Include Controller. I think the project could use some better
> documentation around this and also ship with some examples.
>
> >
> > On the understanding/reading JMeter test plan, my only thought to that
> was
> > good test plan design in terms of renaming components in test plan,
> > labelling them with description of what that component is doing in the
> test
> > and adding in info in the comment fields for components that have them.
> And
> > that would apply to both reviewing test plan in GUI and from parsing JMX
> > file. Any thoughts beyond that?
>
> This is how I do it.
>
> >
> > For the programmatic functionality area, I know we can get that by using
> > JMeter custom plugins, or import Java libraries for use with
> > JUnit/BSF/Beanshell pre/post processors, assertions, and samplers. Build
> > out the needed functionality in code & call it from JMeter, using JMeter
> > for threading, concurrency, load management, and possibly
> reporting/logging
> > too. But any thoughts beyond that? I'm assuming my colleagues were
> looking
> > more for a full featured IDE type tool or language binding based
> framework
> > like Selenium 2/WebDriver where you write in code using "JMeter" APIs to
> do
> > the load testing.
> >
> > Regards,
> > David
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
Re: Some assumptions on JMeter limitations. Your thoughts?
Posted by Anthony Johnson <an...@gmail.com>.
My thoughts below:-)
On Mon, Oct 15, 2012 at 9:26 PM, David Luu <ma...@gmail.com> wrote:
> Hello,
>
> I was asking for feedback on our usage of JMeter within my organization and
> got some responses with some disadvantages/limitations of JMeter. Based on
> the responses, most of it to me seems like some assumptions made w/o
> knowing/researching the full capabilities of JMeter. It took me some time
> to explore and find out JMeter's feature set.
>
> But in any case thought I'd ask the community for input regarding these
> (assumed?) disadvantages, extracted snippet of feedback:
>
> "I don't expect us to use JMeter long-term because it is pretty difficult
> to:
>
> * extract and re-use components, e.g. sign-in
You can create mini-test plans that have a Test Fragment as the base
element and then load those via Include Controllers in the main Test
Plan. This allows you to create all sorts of re-usable components.
> * verify or review the test or your changes to it are doing what you think
> it is doing; you're stuck with either understanding the test through the UI
> or reading jmx directly
Well-Written plans that use the Descriptions and name the components
properly read very well. This is just discipline in using the tool
IMO.
>
> Perhaps you can share some insights in how you have addressed the above
> points.
>
> I think one way to address these issues is to express our performance/load
> tests in a real programming environment. This would give us the normal
> power we expect to manage our test software. There are a few systems that
> allow for this, e.g. gatling [1], grinder [2], iago [3]
>
> [1] http://gatling-tool.org/
> [2] http://grinder.sourceforge.net/
> [3] http://twitter.github.com/iago/
> ...
> Also, having a programming language support gives you a better control on
> the load testing scripts, but Jmeter doesn’t have that."
You can use some Scripting in JMeter, but I purposely chose JMeter to
avoid having to write code. The code barrier sets a baseline for who
can create and understand Performance tests. In my environment, QA
Engineers with no programming experience are able to create, execute
and debug performance tests. Abstracting all the programming with a
usable UI is JMeter's best feature IMO.
>
> The re-use issue seems interesting to me. How best to do that in JMeter?
> For me all I could think of is building a standard test template with
> common components and settings and everyone works off a copy of that
> customizing as needed. But is there a better way to reuse-share a component
> like HTTP header manager, or User Parameters or User Defined Variables, or
> Regular Expression Extractor, etc. with the customized variables, values,
> patterns w/o having to copy paste from one test plan to another?
Include Controller. I think the project could use some better
documentation around this and also ship with some examples.
>
> On the understanding/reading JMeter test plan, my only thought to that was
> good test plan design in terms of renaming components in test plan,
> labelling them with description of what that component is doing in the test
> and adding in info in the comment fields for components that have them. And
> that would apply to both reviewing test plan in GUI and from parsing JMX
> file. Any thoughts beyond that?
This is how I do it.
>
> For the programmatic functionality area, I know we can get that by using
> JMeter custom plugins, or import Java libraries for use with
> JUnit/BSF/Beanshell pre/post processors, assertions, and samplers. Build
> out the needed functionality in code & call it from JMeter, using JMeter
> for threading, concurrency, load management, and possibly reporting/logging
> too. But any thoughts beyond that? I'm assuming my colleagues were looking
> more for a full featured IDE type tool or language binding based framework
> like Selenium 2/WebDriver where you write in code using "JMeter" APIs to do
> the load testing.
>
> Regards,
> David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org