You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Marc Guillemot <mg...@yahoo.fr> on 2007/10/25 16:19:09 UTC
Interest for a combination of JMeter with HtmlUnit / WebTest for
AJAX load testing?
Hello list,
first to present myself: I'm the lead developer of the open source
projects HtmlUnit (http://htmlunit.sf.net) and WebTest
(http://webtest.canoo.com). I'm not a JMeter or load testing expert, so
please forgive my ignorance.
What about a combination JMeter / WebTest (or HtmlUnit)?
An HtmlUnit user has for some months reported that he used HtmlUnit for
load testing his AJAX application and that everything worked smoothly
simulating 350 clients allocating only 512 MB to the JVM. Personally I
haven't tested with so much parallel clients, but my own tests don't
contradict his point. This seems to indicate that HtmlUnit is
lightweight enough to produce load.
Additionally is HtmlUnit a "browser". It is a special kind of browser,
but it is a browser, that tries to behave like FF or IE depending on
what you configure. Particularly it has a pretty good JavaScript
support. As far as I know all (open source as well as commercial) load
testing tools have problems with AJAX because they aren't browsers.
This seems to indicate that HtmlUnit would be currently the only
solution allowing to really produce load as well as to behave like
"normal" browsers on AJAX applications.
Finally, what is possible with HtmlUnit "pure" is probably possible with
WebTest (which itself uses HtmlUnit). The benefit is that it would allow
to specify tests at an higher abstraction level and, in the case of
functional tests, produces great reports.
Now my questions:
- does JMeter's architecture allow to plug something like WebTest or
HtmlUnit as "execution engine"?
- is there interest in JMeter community for such a combination?
Happy testing,
Marc.
--
Blog: http://mguillem.wordpress.com
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
RE: Interest for a combination of JMeter with HtmlUnit / WebTest for AJAX load testing?
Posted by Matthew Coventon <ma...@iseinc.biz>.
I think integration is interesting based on the following:
(1) I have seen interest in this forum for JMeter to have the ability to
process Javascript.
(2) I have seen interest in this forum for JMeter to implement browser
caching behavior ... does HtmlUnit/WebTest implement browser caching
behavior?
Regards,
Matt C.
Matt Coventon
Innovative Software Engineering
mattcoventon@iseinc.biz
Confidentiality Notice: The information contained in this e-mail message
(including any and all attachments or documents linked to this e-mail) may
be legally privileged and confidential information intended only for the use
of the individual or entity named above. If the reader of this message is
not the intended recipient, or the employee or agent responsible to deliver
it to the intended recipient, you are hereby notified that any release,
dissemination, distribution, or copying of this communication is strictly
prohibited. If you have received this communication in error please notify
the author immediately by replying to this message and deleting the original
message. Thank you.
-----Original Message-----
From: news [mailto:news@ger.gmane.org] On Behalf Of Marc Guillemot
Sent: Thursday, October 25, 2007 9:19 AM
To: jmeter-user@jakarta.apache.org
Subject: Interest for a combination of JMeter with HtmlUnit / WebTest for
AJAX load testing?
Hello list,
first to present myself: I'm the lead developer of the open source
projects HtmlUnit (http://htmlunit.sf.net) and WebTest
(http://webtest.canoo.com). I'm not a JMeter or load testing expert, so
please forgive my ignorance.
What about a combination JMeter / WebTest (or HtmlUnit)?
An HtmlUnit user has for some months reported that he used HtmlUnit for
load testing his AJAX application and that everything worked smoothly
simulating 350 clients allocating only 512 MB to the JVM. Personally I
haven't tested with so much parallel clients, but my own tests don't
contradict his point. This seems to indicate that HtmlUnit is
lightweight enough to produce load.
Additionally is HtmlUnit a "browser". It is a special kind of browser,
but it is a browser, that tries to behave like FF or IE depending on
what you configure. Particularly it has a pretty good JavaScript
support. As far as I know all (open source as well as commercial) load
testing tools have problems with AJAX because they aren't browsers.
This seems to indicate that HtmlUnit would be currently the only
solution allowing to really produce load as well as to behave like
"normal" browsers on AJAX applications.
Finally, what is possible with HtmlUnit "pure" is probably possible with
WebTest (which itself uses HtmlUnit). The benefit is that it would allow
to specify tests at an higher abstraction level and, in the case of
functional tests, produces great reports.
Now my questions:
- does JMeter's architecture allow to plug something like WebTest or
HtmlUnit as "execution engine"?
- is there interest in JMeter community for such a combination?
Happy testing,
Marc.
--
Blog: http://mguillem.wordpress.com
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Interest for a combination of JMeter with HtmlUnit / WebTest for AJAX load testing?
Posted by Subrahmanya Bhat <sm...@gmail.com>.
Hey,
I think Dmitry has some valid points. When I used Jmeter for one of retail
projects, the focus was more on loading the server. It was all about the
server gettting bombarded with requests and it responding back.
Plus, I had to maintain the cpu of the load simulating machines well below
50% to make sure that the load simulated is actually happened at the server.
Meaning, had to keep the response processing as low as possible.
Regards
Subrahmanya A
On 10/25/07, Dmitry Kudrenko <dm...@ardas.dp.ua> wrote:
>
>
> MG> Now my questions:
> MG> - does JMeter's architecture allow to plug something like WebTest or
> MG> HtmlUnit as "execution engine"?
> MG> - is there interest in JMeter community for such a combination?
> I am using JMeter and CanooWebTests.
>
> But as for me, there are rather different aims for performance testing
> tool and for functional testing tool.
>
> When you are testing performance you should send requests as quickly
> as possible. You don't need to spend time for parsing html, supporting
> javaScript, building DOM, etc…
>
> You should send requests for loading server. As for AJAX: It is
> possible to test AJAX applications, you don't need support
> HttpXmlRequest for this. Just send request and process the response.
> You need to implement your web client logic on JMeter plan. It is not
> very easy, but it is possible.
>
> Another problem when you want to test application and client should
> send several requests in the separate threads using one session (to
> test AJAX really asynchronous)... But I think it is another theme for
> discussion.
>
> --
> Best regards,
> Dmitry Kudrenko
> ARDAS group http://ardas.dp.ua
> mailto:dmitry@ardas.dp.ua
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>
Re: Interest for a combination of JMeter with HtmlUnit / WebTest for AJAX load testing?
Posted by Dmitry Kudrenko <dm...@ardas.dp.ua>.
MG> Now my questions:
MG> - does JMeter's architecture allow to plug something like WebTest or
MG> HtmlUnit as "execution engine"?
MG> - is there interest in JMeter community for such a combination?
I am using JMeter and CanooWebTests.
But as for me, there are rather different aims for performance testing
tool and for functional testing tool.
When you are testing performance you should send requests as quickly
as possible. You dont need to spend time for parsing html, supporting
javaScript, building DOM, etc
You should send requests for loading server. As for AJAX: It is
possible to test AJAX applications, you dont need support
HttpXmlRequest for this. Just send request and process the response.
You need to implement your web client logic on JMeter plan. It is not
very easy, but it is possible.
Another problem when you want to test application and client should
send several requests in the separate threads using one session (to
test AJAX really asynchronous)... But I think it is another theme for
discussion.
--
Best regards,
Dmitry Kudrenko
ARDAS group http://ardas.dp.ua
mailto:dmitry@ardas.dp.ua
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Interest for a combination of JMeter with HtmlUnit / WebTest
for AJAX load testing?
Posted by Marc Guillemot <mg...@yahoo.fr>.
Hi,
bklau wrote:
> Hi Marc:
> Yes, there is a need to have a lightweight browser simulator.
nice to see someone that understands me ;-)
> I'm seriously considering using HtmlUnit to test Dojo-based app.
hmm, among the "big" AJAX libraries, Dojo is perhaps the one for which
the support is the worst. I've made some fixes last week that now allow
to load Dojo demo email app without problem (simulating FF) but I
haven't had time until now to test more.
> There are 2 main issues that one needs to consider:
> 1. The memory footprint should be small enough to be acceptable.
> We are talking about 500 virtual "browsers", hosted on say, 2-3 load driver
> machines.
the post I referred to mentionned 350 virtual "browsers" in 500MB RAM
for the JVM therefore I can imagine that it is possible to reach 500.
> 2. It is not good enough to send rapid and volumous traffic to the server;
> one need to send the correct kind of traffic pattern;
> otherwise, you have artificial traffic patterns.
> Only a browser emulator can mimick this correctly.
exactly
> I think point #2 is the pehaps the real reason I consider using HtmlUnit vs.
> Http Recordings.
> The reason is that Ajax calls can mingled and interleaved
> non-deterministically.
> At low loads, all client Ajax calls are likely to be serialized; at high
> loads, they are likely to be
> out of sync.
>
> Example: If one have a call pattern sequence ABC..., one would expect reply
> back from server to cluster
> around the same order A'B'C'... (assuming the calls do not do heavy
> processings)
> At high loads, the replys are likely to be out-of-step, maybe
> B'A'C'...unless the Ajax calls within a page("shell") are set to be
> synchronous instead of asynchronous.
> Also, don't forget Ajax client can also dynamically "react" to response from
> earlier Ajax calls sent out previously.
> You see, only a browser "emulator" can simulated the
> Ajax-call-response-react correctly.
>
> To me, Web Test is more a "functional" testing tool; not sure if it is
> suitable for Load Testing even if its build on top of HtmlUnit.
> I'm thinking if you could use an AOP-recorder for HtmlUnit, then you might
> be able to generate high-level code that encapsulate the lower HtmlUnit API
> calls.
> End user can always modify the recording code if necessary.
WebTest exactly provides this high level code and thanks to its Ant
roots you can do a lot of stuff dynamically for instance using own
BuildListener.
Marc.
--
Blog: http://mguillem.wordpress.com
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Interest for a combination of JMeter with HtmlUnit / WebTest for AJAX load testing?
Posted by sebb <se...@gmail.com>.
On 05/11/2007, Marc Guillemot <mg...@yahoo.fr> wrote:
> Hi,
>
> bklau wrote:
> > Hi Marc:
> > Yes, there is a need to have a lightweight browser simulator.
>
> nice to see someone that understands me ;-)
>
> > I'm seriously considering using HtmlUnit to test Dojo-based app.
>
> hmm, among the "big" AJAX libraries, Dojo is perhaps the one for which
> the support is the worst. I've made some fixes last week that now allow
> to load Dojo demo email app without problem (simulating FF) but I
> haven't had time until now to test more.
>
> > There are 2 main issues that one needs to consider:
> > 1. The memory footprint should be small enough to be acceptable.
> > We are talking about 500 virtual "browsers", hosted on say, 2-3 load driver
> > machines.
>
> the post I referred to mentionned 350 virtual "browsers" in 500MB RAM
> for the JVM therefore I can imagine that it is possible to reach 500.
>
> > 2. It is not good enough to send rapid and volumous traffic to the server;
> > one need to send the correct kind of traffic pattern;
> > otherwise, you have artificial traffic patterns.
> > Only a browser emulator can mimick this correctly.
>
> exactly
Note that different browsers behave in different ways - indeed
different versions of the same browser probably do too, and anyway
users can tweak the settings that affect the browser behaviour.
So exact emulation is not necessarily an achievable aim, nor even a
necessary aim.
> > I think point #2 is the pehaps the real reason I consider using HtmlUnit vs.
> > Http Recordings.
> > The reason is that Ajax calls can mingled and interleaved
> > non-deterministically.
> > At low loads, all client Ajax calls are likely to be serialized; at high
> > loads, they are likely to be
> > out of sync.
> >
> > Example: If one have a call pattern sequence ABC..., one would expect reply
> > back from server to cluster
> > around the same order A'B'C'... (assuming the calls do not do heavy
> > processings)
> > At high loads, the replys are likely to be out-of-step, maybe
> > B'A'C'...unless the Ajax calls within a page("shell") are set to be
> > synchronous instead of asynchronous.
> > Also, don't forget Ajax client can also dynamically "react" to response from
> > earlier Ajax calls sent out previously.
> > You see, only a browser "emulator" can simulated the
> > Ajax-call-response-react correctly.
> >
> > To me, Web Test is more a "functional" testing tool; not sure if it is
> > suitable for Load Testing even if its build on top of HtmlUnit.
> > I'm thinking if you could use an AOP-recorder for HtmlUnit, then you might
> > be able to generate high-level code that encapsulate the lower HtmlUnit API
> > calls.
> > End user can always modify the recording code if necessary.
>
> WebTest exactly provides this high level code and thanks to its Ant
> roots you can do a lot of stuff dynamically for instance using own
> BuildListener.
>
> Marc.
> --
> Blog: http://mguillem.wordpress.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Interest for a combination of JMeter with HtmlUnit / WebTest
for AJAX load testing?
Posted by Marc Guillemot <mg...@yahoo.fr>.
Hi,
bklau wrote:
> Hi Marc:
> Yes, there is a need to have a lightweight browser simulator.
nice to see someone that understands me ;-)
> I'm seriously considering using HtmlUnit to test Dojo-based app.
hmm, among the "big" AJAX libraries, Dojo is perhaps the one for which
the support is the worst. I've made some fixes last week that now allow
to load Dojo demo email app without problem (simulating FF) but I
haven't had time until now to test more.
> There are 2 main issues that one needs to consider:
> 1. The memory footprint should be small enough to be acceptable.
> We are talking about 500 virtual "browsers", hosted on say, 2-3 load driver
> machines.
the post I referred to mentionned 350 virtual "browsers" in 500MB RAM
for the JVM therefore I can imagine that it is possible to reach 500.
> 2. It is not good enough to send rapid and volumous traffic to the server;
> one need to send the correct kind of traffic pattern;
> otherwise, you have artificial traffic patterns.
> Only a browser emulator can mimick this correctly.
exactly
> I think point #2 is the pehaps the real reason I consider using HtmlUnit vs.
> Http Recordings.
> The reason is that Ajax calls can mingled and interleaved
> non-deterministically.
> At low loads, all client Ajax calls are likely to be serialized; at high
> loads, they are likely to be
> out of sync.
>
> Example: If one have a call pattern sequence ABC..., one would expect reply
> back from server to cluster
> around the same order A'B'C'... (assuming the calls do not do heavy
> processings)
> At high loads, the replys are likely to be out-of-step, maybe
> B'A'C'...unless the Ajax calls within a page("shell") are set to be
> synchronous instead of asynchronous.
> Also, don't forget Ajax client can also dynamically "react" to response from
> earlier Ajax calls sent out previously.
> You see, only a browser "emulator" can simulated the
> Ajax-call-response-react correctly.
>
> To me, Web Test is more a "functional" testing tool; not sure if it is
> suitable for Load Testing even if its build on top of HtmlUnit.
> I'm thinking if you could use an AOP-recorder for HtmlUnit, then you might
> be able to generate high-level code that encapsulate the lower HtmlUnit API
> calls.
> End user can always modify the recording code if necessary.
WebTest exactly provides this high level code and thanks to its Ant
roots you can do a lot of stuff dynamically for instance using own
BuildListener.
Marc.
--
Blog: http://mguillem.wordpress.com
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Interest for a combination of JMeter with HtmlUnit / WebTest
for AJAX load testing?
Posted by bklau <bk...@gmail.com>.
Hi Marc:
Yes, there is a need to have a lightweight browser simulator.
I'm seriously considering using HtmlUnit to test Dojo-based app.
There are 2 main issues that one needs to consider:
1. The memory footprint should be small enough to be acceptable.
We are talking about 500 virtual "browsers", hosted on say, 2-3 load driver
machines.
2. It is not good enough to send rapid and volumous traffic to the server;
one need to send the correct kind of traffic pattern;
otherwise, you have artificial traffic patterns.
Only a browser emulator can mimick this correctly.
I think point #2 is the pehaps the real reason I consider using HtmlUnit vs.
Http Recordings.
The reason is that Ajax calls can mingled and interleaved
non-deterministically.
At low loads, all client Ajax calls are likely to be serialized; at high
loads, they are likely to be
out of sync.
Example: If one have a call pattern sequence ABC..., one would expect reply
back from server to cluster
around the same order A'B'C'... (assuming the calls do not do heavy
processings)
At high loads, the replys are likely to be out-of-step, maybe
B'A'C'...unless the Ajax calls within a page("shell") are set to be
synchronous instead of asynchronous.
Also, don't forget Ajax client can also dynamically "react" to response from
earlier Ajax calls sent out previously.
You see, only a browser "emulator" can simulated the
Ajax-call-response-react correctly.
To me, Web Test is more a "functional" testing tool; not sure if it is
suitable for Load Testing even if its build on top of HtmlUnit.
I'm thinking if you could use an AOP-recorder for HtmlUnit, then you might
be able to generate high-level code that encapsulate the lower HtmlUnit API
calls.
End user can always modify the recording code if necessary.
/BK
Marc Guillemot wrote:
>
> Hello list,
>
> first to present myself: I'm the lead developer of the open source
> projects HtmlUnit (http://htmlunit.sf.net) and WebTest
> (http://webtest.canoo.com). I'm not a JMeter or load testing expert, so
> please forgive my ignorance.
>
>
> What about a combination JMeter / WebTest (or HtmlUnit)?
>
>
> An HtmlUnit user has for some months reported that he used HtmlUnit for
> load testing his AJAX application and that everything worked smoothly
> simulating 350 clients allocating only 512 MB to the JVM. Personally I
> haven't tested with so much parallel clients, but my own tests don't
> contradict his point. This seems to indicate that HtmlUnit is
> lightweight enough to produce load.
>
> Additionally is HtmlUnit a "browser". It is a special kind of browser,
> but it is a browser, that tries to behave like FF or IE depending on
> what you configure. Particularly it has a pretty good JavaScript
> support. As far as I know all (open source as well as commercial) load
> testing tools have problems with AJAX because they aren't browsers.
>
> This seems to indicate that HtmlUnit would be currently the only
> solution allowing to really produce load as well as to behave like
> "normal" browsers on AJAX applications.
>
> Finally, what is possible with HtmlUnit "pure" is probably possible with
> WebTest (which itself uses HtmlUnit). The benefit is that it would allow
> to specify tests at an higher abstraction level and, in the case of
> functional tests, produces great reports.
>
> Now my questions:
> - does JMeter's architecture allow to plug something like WebTest or
> HtmlUnit as "execution engine"?
> - is there interest in JMeter community for such a combination?
>
> Happy testing,
> Marc.
> --
> Blog: http://mguillem.wordpress.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Interest-for-a-combination-of-JMeter-with-HtmlUnit---WebTest-for-AJAX-load-testing--tf4691247.html#a13547598
Sent from the JMeter - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org