You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by Cheen-Pin Lim <ch...@gmail.com> on 2012/04/21 08:55:39 UTC
Web Driver based Sampler?
Hi,
After looking at the current HTTP Sampler, I would like to propose a
WebDriver based implementation. The problem is that with a lot of modern
websites using AJAX, advanced CSS3 (transitions/transforms) to do the
rendering, the performance problem may not be 'visible' to the standard
HTTP Sampler. The idea is that if we could use JMeter to control a browser
and collect the time it takes to render each 'page' it would give the users
an understanding of how long a page would take to render.
Some of the high level goals are as follows:
1. Allow any WebDriver supported browser to be used to perform the
Sampling.
2. Provide a BeanShell/BSF style pane where the user can script the
behaviour of WebDriver.
I've got a very basic implementation working here:
https://github.com/cplim/JMeter - you'll have to switch to the 'webmeter'
branch to see the changes. There are limitations with this proof of
concept, mainly:
1. It only creates Firefox instances
2. I've only provided Javascript lang support using BSF.
Some outstanding questions:
1. How to support Phone (Android/iOS) based WebDriver, as these seem to
expect to only connect to a single phone.
I would be happy to contribute the changes back to JMeter. Can I get some
feedback on this proposal, and what I need to do to contribute my changes
back to the main JMeter codebase?
regards,
CP Lim
Re: Web Driver based Sampler?
Posted by Shmuel Krakower <sh...@gmail.com>.
Hi
This is something to discuss. HP implemented this in their latest version
of load runner with the new protocol which I cannot recall it's name.
I do find it useful to allow more close to reality load testing.
On the other hand, it is absolutly not mandatory as load testing is all
about loading the system under test, but may be a stepping stone.
One thing which I read is that webdriver is not very accurate on time
stamping so this may be a pain in the ass to try and use it to provide
accurate response times.
Other toughts from the community?
Shmuel.
בתאריך 2012 4 21 09:56, מאת "Cheen-Pin Lim" <ch...@gmail.com>:
> Hi,
>
> After looking at the current HTTP Sampler, I would like to propose a
> WebDriver based implementation. The problem is that with a lot of modern
> websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> rendering, the performance problem may not be 'visible' to the standard
> HTTP Sampler. The idea is that if we could use JMeter to control a browser
> and collect the time it takes to render each 'page' it would give the users
> an understanding of how long a page would take to render.
>
> Some of the high level goals are as follows:
>
> 1. Allow any WebDriver supported browser to be used to perform the
> Sampling.
> 2. Provide a BeanShell/BSF style pane where the user can script the
> behaviour of WebDriver.
>
> I've got a very basic implementation working here:
> https://github.com/cplim/JMeter - you'll have to switch to the 'webmeter'
> branch to see the changes. There are limitations with this proof of
> concept, mainly:
>
> 1. It only creates Firefox instances
> 2. I've only provided Javascript lang support using BSF.
>
> Some outstanding questions:
>
> 1. How to support Phone (Android/iOS) based WebDriver, as these seem to
> expect to only connect to a single phone.
>
> I would be happy to contribute the changes back to JMeter. Can I get some
> feedback on this proposal, and what I need to do to contribute my changes
> back to the main JMeter codebase?
>
> regards,
> CP Lim
>
Re: Web Driver based Sampler?
Posted by Cheen-Pin Lim <ch...@gmail.com>.
Hi Alon,
Thanks for the feedback, glad you like the idea. I have an early prototype
working already here: https://github.com/cplim/JMeter/tree/webmeter (its
basically under the webmeter branch right now).
As mentioned, it is still in the early stages, but things I already have
working:
1. Created a 'Web Browser' Sampler implementation that will fire up a
browser.
2. Starts a (Firefox) browser per thread, and shuts it down after the
script finishes.
3. Integrated Javascript BSF so that the browser for that thread can be
accessed and scripted to perform a sequence of actions.
If you do clone the git repo, you'll have to switch to the 'webmeter'
branch and run the following ant commands:
1. ant download_jars
2. ant run_gui
In there you should be able to create a 'Web Sampler' and control Firefox
via the exposed bean called 'browser'. You can just type:
=============
browser.get('http://www.google.com')
=============
And it should work. :)
I agree, the Mobile browsers are probably more challenging, so yes my
immediate focus is just on desktop browsers. I have yet to implement the
other Browsers and get then use this in anger. :)
Any further feedback would be welcome.
regards,
CP
On 21 April 2012 22:06, Alon Girmonsky <al...@blazemeter.com> wrote:
> Hi Lim.
>
> This sounds like an excellent idea. I also feel that this part is missing
> from JMeter.
> I have a few comments:
> 1. You should consider writing a plugin which can maybe be more simple to
> start with.
> 2. Smartphone will be a small challenge compared to real browsers. Once
> that is solve, smartphones implementation will be easy.
> 3. The tool I know that can simulate (automate) a browser is Selenium. Yet,
> it looks like a challenge (probably unlikely) to implement it into JMeter.
>
> As I mentioned it sounds great and required.
>
> Alon.
> http://blazemeter.com/
>
>
> On Sat, Apr 21, 2012 at 2:07 PM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi,
> >
> > The tools mentioned help a great deal, however they generally look at it
> > from a single browser to a single server scenario. Once you have more
> load
> > applied to the servers/services different characteristics tend to emerge,
> > which is why I was thinking that the use of JMeter would be useful.
> >
> > Furthermore, if our primary audience is hitting the site using a browser,
> > then would it not make sense to use the same tool to simulate the load?
> >
> > Just my thoughts.
> >
> > regards,
> > CP
> >
> > On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com>
> wrote:
> >
> > > Pleanty of awesome tools are already available for free to meet your
> > goals.
> > >
> > > Free:
> > > https://developers.google.com/pagespeed/
> > > http://yslow.org/
> > >
> > > Free or Premium:
> > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > >
> > >
> > > Selenium has a webdriver which you can plan on extending - if you will.
> > > IMHO Jmeter is not the right platform for this.
> > > Thanks
> > > Chaitnaya M Bhatt
> > > http://www.performancecompetence.com
> > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> cheenpin.lim@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > After looking at the current HTTP Sampler, I would like to propose a
> > > > WebDriver based implementation. The problem is that with a lot of
> > modern
> > > > websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> > > > rendering, the performance problem may not be 'visible' to the
> standard
> > > > HTTP Sampler. The idea is that if we could use JMeter to control a
> > > browser
> > > > and collect the time it takes to render each 'page' it would give the
> > > users
> > > > an understanding of how long a page would take to render.
> > > >
> > > > Some of the high level goals are as follows:
> > > >
> > > > 1. Allow any WebDriver supported browser to be used to perform the
> > > > Sampling.
> > > > 2. Provide a BeanShell/BSF style pane where the user can script the
> > > > behaviour of WebDriver.
> > > >
> > > > I've got a very basic implementation working here:
> > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > 'webmeter'
> > > > branch to see the changes. There are limitations with this proof of
> > > > concept, mainly:
> > > >
> > > > 1. It only creates Firefox instances
> > > > 2. I've only provided Javascript lang support using BSF.
> > > >
> > > > Some outstanding questions:
> > > >
> > > > 1. How to support Phone (Android/iOS) based WebDriver, as these
> seem
> > to
> > > > expect to only connect to a single phone.
> > > >
> > > > I would be happy to contribute the changes back to JMeter. Can I get
> > > some
> > > > feedback on this proposal, and what I need to do to contribute my
> > changes
> > > > back to the main JMeter codebase?
> > > >
> > > > regards,
> > > > CP Lim
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Alon Girmonsky <al...@blazemeter.com>.
Hi Lim.
This sounds like an excellent idea. I also feel that this part is missing
from JMeter.
I have a few comments:
1. You should consider writing a plugin which can maybe be more simple to
start with.
2. Smartphone will be a small challenge compared to real browsers. Once
that is solve, smartphones implementation will be easy.
3. The tool I know that can simulate (automate) a browser is Selenium. Yet,
it looks like a challenge (probably unlikely) to implement it into JMeter.
As I mentioned it sounds great and required.
Alon.
http://blazemeter.com/
On Sat, Apr 21, 2012 at 2:07 PM, Cheen-Pin Lim <ch...@gmail.com>wrote:
> Hi,
>
> The tools mentioned help a great deal, however they generally look at it
> from a single browser to a single server scenario. Once you have more load
> applied to the servers/services different characteristics tend to emerge,
> which is why I was thinking that the use of JMeter would be useful.
>
> Furthermore, if our primary audience is hitting the site using a browser,
> then would it not make sense to use the same tool to simulate the load?
>
> Just my thoughts.
>
> regards,
> CP
>
> On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com> wrote:
>
> > Pleanty of awesome tools are already available for free to meet your
> goals.
> >
> > Free:
> > https://developers.google.com/pagespeed/
> > http://yslow.org/
> >
> > Free or Premium:
> > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> >
> >
> > Selenium has a webdriver which you can plan on extending - if you will.
> > IMHO Jmeter is not the right platform for this.
> > Thanks
> > Chaitnaya M Bhatt
> > http://www.performancecompetence.com
> > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > After looking at the current HTTP Sampler, I would like to propose a
> > > WebDriver based implementation. The problem is that with a lot of
> modern
> > > websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> > > rendering, the performance problem may not be 'visible' to the standard
> > > HTTP Sampler. The idea is that if we could use JMeter to control a
> > browser
> > > and collect the time it takes to render each 'page' it would give the
> > users
> > > an understanding of how long a page would take to render.
> > >
> > > Some of the high level goals are as follows:
> > >
> > > 1. Allow any WebDriver supported browser to be used to perform the
> > > Sampling.
> > > 2. Provide a BeanShell/BSF style pane where the user can script the
> > > behaviour of WebDriver.
> > >
> > > I've got a very basic implementation working here:
> > > https://github.com/cplim/JMeter - you'll have to switch to the
> > 'webmeter'
> > > branch to see the changes. There are limitations with this proof of
> > > concept, mainly:
> > >
> > > 1. It only creates Firefox instances
> > > 2. I've only provided Javascript lang support using BSF.
> > >
> > > Some outstanding questions:
> > >
> > > 1. How to support Phone (Android/iOS) based WebDriver, as these seem
> to
> > > expect to only connect to a single phone.
> > >
> > > I would be happy to contribute the changes back to JMeter. Can I get
> > some
> > > feedback on this proposal, and what I need to do to contribute my
> changes
> > > back to the main JMeter codebase?
> > >
> > > regards,
> > > CP Lim
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Cheen-Pin Lim <ch...@gmail.com>.
Hi Shmuel,
My initial thoughts on this is to perform this in separate samplers. If
the user wishes to do the following:
1. Go to Google.com
2. Perform a search for cheese
3. Click on the first result.
I would represent this with 3 Samplers for each of the steps above. This
way we can track how long each of the pages (home page, search results,
first search result) by adding a Listener to each of the samplers.
This way I accomplished this is by re-using the same WebDriver (browser)
instance between each of the Samplers, and storing this in a ThreadlLocal
variable.
Hope this answers your question. If not, I can send you a JMX file that
describes this behaviour (can send it to your gmail account instead of
spamming this list).
regards,
CP
On 7 May 2012 22:30, Shmuel Krakower <sh...@gmail.com> wrote:
> Hi CP,
> After looking at the sampler - one question in mind:
> I might want to get more details on specific action inside the script, i.e:
> browse into google.com
> search for a value
>
> I'd like to know how much time the search took - and I can't find an easy
> solution for this.
> Any ideas?
>
> Regards,
> Shmuel.
>
> On Mon, Apr 23, 2012 at 3:38 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi Shmuel,
> >
> > I've yet to try the WebSampler implementation on our test environment.
> > Unfortunately I'm away for the next 2 weeks, but when I get back to
> work I
> > can try out the WebSampler and see if I can replicate the statistics I'm
> > getting in our production environment.
> >
> > regards,
> > CP
> >
> > On 22 April 2012 23:49, Shmuel Krakower <sh...@gmail.com> wrote:
> >
> > > I'd suggest to HP to drop C language scripting in order to make
> scripting
> > > easier / shorter.
> > >
> > > As Chaitanya said - browser response times is overrated.
> > >
> > > Anyhow, I agree that implementing load tests with hundreds / thousands
> of
> > > virtual users with that kind of a sampler(which running a browser) will
> > > require much more hardware but as CP said, with today's cloud services
> > this
> > > is very affordable, thus this is not the issue.
> > > I think that there are services on the web which already doing this (
> > > https://browsermob.com/website-load-testing-features), but I never
> tried
> > > those.
> > >
> > > CP, if you get different numbers between you monitoring solution and
> > JMeter
> > > reports, you should try to understand why that happens.
> > > It might be for many reasons. Did developing this sampler resolved your
> > > original problem? Do you get more close numbers now?
> > >
> > > Shmuel.
> > >
> > >
> > > On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > > >wrote:
> > >
> > > > Hi Chaitanya,
> > > >
> > > > I guess I am looking at this from a browser view rather than from the
> > > > server side. What I tend to find is that current load testing does
> not
> > > > properly reflect the browser load times on our site. The reports
> from
> > > > Google (in webmasters tools) and newrelic.com give a very different
> > > > picture
> > > > from that done during load/performance testing.
> > > >
> > > > We use HP Load Runner for our in-house load testing and the numbers
> we
> > > get
> > > > (even when we download 'all' external resources) does not come close
> to
> > > the
> > > > numbers reported by the other tools. I guess I am attempting to
> > address
> > > a
> > > > gap that I see in our in-house load/performance testing, and the
> > > > integration of webdriver into JMeter could solve this problem. At
> this
> > > > stage it is an attempt to try to provide a means to simulate what our
> > > > production servers see (i.e. using a real browser that will spend
> time
> > > > executing JS/CSS).
> > > >
> > > > Yes, I agree that the complexity and cost of running a browser
> instance
> > > is
> > > > higher than using the standard HTTP Sampler, but with modern services
> > > such
> > > > as Amazon AWS, I would think that these problems would be
> > > > addressable/solvable.
> > > >
> > > > regards,
> > > > CP
> > > >
> > > > On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com>
> > > wrote:
> > > >
> > > > > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > > cheenpin.lim@gmail.com
> > > > > >wrote:
> > > > > >Once you have more load
> > > > > >applied to the servers/services different characteristics tend to
> > > > emerge,
> > > > > >which is why I was thinking that the use of JMeter would be
> useful.
> > > > >
> > > > > *I may totally sound cynical while saying this but honestly the
> > reason
> > > > why
> > > > > I am criticising your idea is mainly due to the week rational
> behind
> > > your
> > > > > motivation to create this feature:
> > > > >
> > > > > 1. First of all browser induced latency on the total workload is
> > > grossly
> > > > > overrated. Having said that, all that an accurate simulation of
> > browser
> > > > > latency would do to your workload is that it would marginally
> reduce
> > > the
> > > > > total hits made on the server, which I think is bad from a
> > fundamental
> > > > load
> > > > > testing perspective.
> > > > > 2.To accommodate in-memory containers to hold 100s-1000s of virutal
> > > user
> > > > > browser sessions is practically impossible without testers having
> to
> > > > > significantly size up their test lab resources. Think about the
> cost
> > > > > factor.
> > > > > 3. Looking at todays trend of browser adoption your web driver
> would
> > > have
> > > > > to be equipped with simulation capability multiple types of
> browsers
> > > such
> > > > > as IE, Chromer, Safari etc. etc. and also you would have to
> consider
> > > > > simulating various device characteristics on the generated load.
> > Think
> > > > > about the complexity.
> > > > >
> > > > > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > > > > TrueClient. HP came out with Ajax TrueClient not with an intention
> > > > > of generating real browser traffic but it was mainly developed to
> > > reduce
> > > > > the complexity of scripting apps that employ unintelligible data
> > > > > serialization like Google GWT. With this HP's protocol you wouldn't
> > > have
> > > > a
> > > > > necessity of correlating session variables at all and there by you
> > > would
> > > > be
> > > > > significantly reducing the time to develop scripts. However, this
> > > > protocol
> > > > > currently has poor user adoption due to various limitations.
> > > > >
> > > > > Thanks
> > > > > Chaitanya M Bhatt
> > > > > http://www.performancecompetence.com
> > > > >
> > > > > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > cheenpin.lim@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > The tools mentioned help a great deal, however they generally
> look
> > at
> > > > it
> > > > > > from a single browser to a single server scenario. Once you have
> > > more
> > > > > load
> > > > > > applied to the servers/services different characteristics tend to
> > > > emerge,
> > > > > > which is why I was thinking that the use of JMeter would be
> useful.
> > > > > >
> > > > > > Furthermore, if our primary audience is hitting the site using a
> > > > browser,
> > > > > > then would it not make sense to use the same tool to simulate the
> > > load?
> > > > > >
> > > > > > Just my thoughts.
> > > > > >
> > > > > > regards,
> > > > > > CP
> > > > > >
> > > > > > On 21 April 2012 17:26, chaitanya bhatt <
> bhatt.chaitanya@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Pleanty of awesome tools are already available for free to meet
> > > your
> > > > > > goals.
> > > > > > >
> > > > > > > Free:
> > > > > > > https://developers.google.com/pagespeed/
> > > > > > > http://yslow.org/
> > > > > > >
> > > > > > > Free or Premium:
> > > > > > >
> http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > > > > >
> > > > > > >
> > > > > > > Selenium has a webdriver which you can plan on extending - if
> you
> > > > will.
> > > > > > > IMHO Jmeter is not the right platform for this.
> > > > > > > Thanks
> > > > > > > Chaitnaya M Bhatt
> > > > > > > http://www.performancecompetence.com
> > > > > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > > > > cheenpin.lim@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > After looking at the current HTTP Sampler, I would like to
> > > propose
> > > > a
> > > > > > > > WebDriver based implementation. The problem is that with a
> lot
> > > of
> > > > > > modern
> > > > > > > > websites using AJAX, advanced CSS3 (transitions/transforms)
> to
> > do
> > > > the
> > > > > > > > rendering, the performance problem may not be 'visible' to
> the
> > > > > standard
> > > > > > > > HTTP Sampler. The idea is that if we could use JMeter to
> > > control a
> > > > > > > browser
> > > > > > > > and collect the time it takes to render each 'page' it would
> > give
> > > > the
> > > > > > > users
> > > > > > > > an understanding of how long a page would take to render.
> > > > > > > >
> > > > > > > > Some of the high level goals are as follows:
> > > > > > > >
> > > > > > > > 1. Allow any WebDriver supported browser to be used to
> > perform
> > > > the
> > > > > > > > Sampling.
> > > > > > > > 2. Provide a BeanShell/BSF style pane where the user can
> > script
> > > > the
> > > > > > > > behaviour of WebDriver.
> > > > > > > >
> > > > > > > > I've got a very basic implementation working here:
> > > > > > > > https://github.com/cplim/JMeter - you'll have to switch to
> the
> > > > > > > 'webmeter'
> > > > > > > > branch to see the changes. There are limitations with this
> > proof
> > > > of
> > > > > > > > concept, mainly:
> > > > > > > >
> > > > > > > > 1. It only creates Firefox instances
> > > > > > > > 2. I've only provided Javascript lang support using BSF.
> > > > > > > >
> > > > > > > > Some outstanding questions:
> > > > > > > >
> > > > > > > > 1. How to support Phone (Android/iOS) based WebDriver, as
> > these
> > > > > seem
> > > > > > to
> > > > > > > > expect to only connect to a single phone.
> > > > > > > >
> > > > > > > > I would be happy to contribute the changes back to JMeter.
> > Can I
> > > > get
> > > > > > > some
> > > > > > > > feedback on this proposal, and what I need to do to
> contribute
> > my
> > > > > > changes
> > > > > > > > back to the main JMeter codebase?
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > CP Lim
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Shmuel Krakower <sh...@gmail.com>.
Hi CP,
After looking at the sampler - one question in mind:
I might want to get more details on specific action inside the script, i.e:
browse into google.com
search for a value
I'd like to know how much time the search took - and I can't find an easy
solution for this.
Any ideas?
Regards,
Shmuel.
On Mon, Apr 23, 2012 at 3:38 AM, Cheen-Pin Lim <ch...@gmail.com>wrote:
> Hi Shmuel,
>
> I've yet to try the WebSampler implementation on our test environment.
> Unfortunately I'm away for the next 2 weeks, but when I get back to work I
> can try out the WebSampler and see if I can replicate the statistics I'm
> getting in our production environment.
>
> regards,
> CP
>
> On 22 April 2012 23:49, Shmuel Krakower <sh...@gmail.com> wrote:
>
> > I'd suggest to HP to drop C language scripting in order to make scripting
> > easier / shorter.
> >
> > As Chaitanya said - browser response times is overrated.
> >
> > Anyhow, I agree that implementing load tests with hundreds / thousands of
> > virtual users with that kind of a sampler(which running a browser) will
> > require much more hardware but as CP said, with today's cloud services
> this
> > is very affordable, thus this is not the issue.
> > I think that there are services on the web which already doing this (
> > https://browsermob.com/website-load-testing-features), but I never tried
> > those.
> >
> > CP, if you get different numbers between you monitoring solution and
> JMeter
> > reports, you should try to understand why that happens.
> > It might be for many reasons. Did developing this sampler resolved your
> > original problem? Do you get more close numbers now?
> >
> > Shmuel.
> >
> >
> > On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> >
> > > Hi Chaitanya,
> > >
> > > I guess I am looking at this from a browser view rather than from the
> > > server side. What I tend to find is that current load testing does not
> > > properly reflect the browser load times on our site. The reports from
> > > Google (in webmasters tools) and newrelic.com give a very different
> > > picture
> > > from that done during load/performance testing.
> > >
> > > We use HP Load Runner for our in-house load testing and the numbers we
> > get
> > > (even when we download 'all' external resources) does not come close to
> > the
> > > numbers reported by the other tools. I guess I am attempting to
> address
> > a
> > > gap that I see in our in-house load/performance testing, and the
> > > integration of webdriver into JMeter could solve this problem. At this
> > > stage it is an attempt to try to provide a means to simulate what our
> > > production servers see (i.e. using a real browser that will spend time
> > > executing JS/CSS).
> > >
> > > Yes, I agree that the complexity and cost of running a browser instance
> > is
> > > higher than using the standard HTTP Sampler, but with modern services
> > such
> > > as Amazon AWS, I would think that these problems would be
> > > addressable/solvable.
> > >
> > > regards,
> > > CP
> > >
> > > On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com>
> > wrote:
> > >
> > > > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > cheenpin.lim@gmail.com
> > > > >wrote:
> > > > >Once you have more load
> > > > >applied to the servers/services different characteristics tend to
> > > emerge,
> > > > >which is why I was thinking that the use of JMeter would be useful.
> > > >
> > > > *I may totally sound cynical while saying this but honestly the
> reason
> > > why
> > > > I am criticising your idea is mainly due to the week rational behind
> > your
> > > > motivation to create this feature:
> > > >
> > > > 1. First of all browser induced latency on the total workload is
> > grossly
> > > > overrated. Having said that, all that an accurate simulation of
> browser
> > > > latency would do to your workload is that it would marginally reduce
> > the
> > > > total hits made on the server, which I think is bad from a
> fundamental
> > > load
> > > > testing perspective.
> > > > 2.To accommodate in-memory containers to hold 100s-1000s of virutal
> > user
> > > > browser sessions is practically impossible without testers having to
> > > > significantly size up their test lab resources. Think about the cost
> > > > factor.
> > > > 3. Looking at todays trend of browser adoption your web driver would
> > have
> > > > to be equipped with simulation capability multiple types of browsers
> > such
> > > > as IE, Chromer, Safari etc. etc. and also you would have to consider
> > > > simulating various device characteristics on the generated load.
> Think
> > > > about the complexity.
> > > >
> > > > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > > > TrueClient. HP came out with Ajax TrueClient not with an intention
> > > > of generating real browser traffic but it was mainly developed to
> > reduce
> > > > the complexity of scripting apps that employ unintelligible data
> > > > serialization like Google GWT. With this HP's protocol you wouldn't
> > have
> > > a
> > > > necessity of correlating session variables at all and there by you
> > would
> > > be
> > > > significantly reducing the time to develop scripts. However, this
> > > protocol
> > > > currently has poor user adoption due to various limitations.
> > > >
> > > > Thanks
> > > > Chaitanya M Bhatt
> > > > http://www.performancecompetence.com
> > > >
> > > > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> cheenpin.lim@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > The tools mentioned help a great deal, however they generally look
> at
> > > it
> > > > > from a single browser to a single server scenario. Once you have
> > more
> > > > load
> > > > > applied to the servers/services different characteristics tend to
> > > emerge,
> > > > > which is why I was thinking that the use of JMeter would be useful.
> > > > >
> > > > > Furthermore, if our primary audience is hitting the site using a
> > > browser,
> > > > > then would it not make sense to use the same tool to simulate the
> > load?
> > > > >
> > > > > Just my thoughts.
> > > > >
> > > > > regards,
> > > > > CP
> > > > >
> > > > > On 21 April 2012 17:26, chaitanya bhatt <bhatt.chaitanya@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Pleanty of awesome tools are already available for free to meet
> > your
> > > > > goals.
> > > > > >
> > > > > > Free:
> > > > > > https://developers.google.com/pagespeed/
> > > > > > http://yslow.org/
> > > > > >
> > > > > > Free or Premium:
> > > > > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > > > >
> > > > > >
> > > > > > Selenium has a webdriver which you can plan on extending - if you
> > > will.
> > > > > > IMHO Jmeter is not the right platform for this.
> > > > > > Thanks
> > > > > > Chaitnaya M Bhatt
> > > > > > http://www.performancecompetence.com
> > > > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > > > cheenpin.lim@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > After looking at the current HTTP Sampler, I would like to
> > propose
> > > a
> > > > > > > WebDriver based implementation. The problem is that with a lot
> > of
> > > > > modern
> > > > > > > websites using AJAX, advanced CSS3 (transitions/transforms) to
> do
> > > the
> > > > > > > rendering, the performance problem may not be 'visible' to the
> > > > standard
> > > > > > > HTTP Sampler. The idea is that if we could use JMeter to
> > control a
> > > > > > browser
> > > > > > > and collect the time it takes to render each 'page' it would
> give
> > > the
> > > > > > users
> > > > > > > an understanding of how long a page would take to render.
> > > > > > >
> > > > > > > Some of the high level goals are as follows:
> > > > > > >
> > > > > > > 1. Allow any WebDriver supported browser to be used to
> perform
> > > the
> > > > > > > Sampling.
> > > > > > > 2. Provide a BeanShell/BSF style pane where the user can
> script
> > > the
> > > > > > > behaviour of WebDriver.
> > > > > > >
> > > > > > > I've got a very basic implementation working here:
> > > > > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > > > > 'webmeter'
> > > > > > > branch to see the changes. There are limitations with this
> proof
> > > of
> > > > > > > concept, mainly:
> > > > > > >
> > > > > > > 1. It only creates Firefox instances
> > > > > > > 2. I've only provided Javascript lang support using BSF.
> > > > > > >
> > > > > > > Some outstanding questions:
> > > > > > >
> > > > > > > 1. How to support Phone (Android/iOS) based WebDriver, as
> these
> > > > seem
> > > > > to
> > > > > > > expect to only connect to a single phone.
> > > > > > >
> > > > > > > I would be happy to contribute the changes back to JMeter.
> Can I
> > > get
> > > > > > some
> > > > > > > feedback on this proposal, and what I need to do to contribute
> my
> > > > > changes
> > > > > > > back to the main JMeter codebase?
> > > > > > >
> > > > > > > regards,
> > > > > > > CP Lim
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Cheen-Pin Lim <ch...@gmail.com>.
Hi Shmuel,
I've yet to try the WebSampler implementation on our test environment.
Unfortunately I'm away for the next 2 weeks, but when I get back to work I
can try out the WebSampler and see if I can replicate the statistics I'm
getting in our production environment.
regards,
CP
On 22 April 2012 23:49, Shmuel Krakower <sh...@gmail.com> wrote:
> I'd suggest to HP to drop C language scripting in order to make scripting
> easier / shorter.
>
> As Chaitanya said - browser response times is overrated.
>
> Anyhow, I agree that implementing load tests with hundreds / thousands of
> virtual users with that kind of a sampler(which running a browser) will
> require much more hardware but as CP said, with today's cloud services this
> is very affordable, thus this is not the issue.
> I think that there are services on the web which already doing this (
> https://browsermob.com/website-load-testing-features), but I never tried
> those.
>
> CP, if you get different numbers between you monitoring solution and JMeter
> reports, you should try to understand why that happens.
> It might be for many reasons. Did developing this sampler resolved your
> original problem? Do you get more close numbers now?
>
> Shmuel.
>
>
> On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi Chaitanya,
> >
> > I guess I am looking at this from a browser view rather than from the
> > server side. What I tend to find is that current load testing does not
> > properly reflect the browser load times on our site. The reports from
> > Google (in webmasters tools) and newrelic.com give a very different
> > picture
> > from that done during load/performance testing.
> >
> > We use HP Load Runner for our in-house load testing and the numbers we
> get
> > (even when we download 'all' external resources) does not come close to
> the
> > numbers reported by the other tools. I guess I am attempting to address
> a
> > gap that I see in our in-house load/performance testing, and the
> > integration of webdriver into JMeter could solve this problem. At this
> > stage it is an attempt to try to provide a means to simulate what our
> > production servers see (i.e. using a real browser that will spend time
> > executing JS/CSS).
> >
> > Yes, I agree that the complexity and cost of running a browser instance
> is
> > higher than using the standard HTTP Sampler, but with modern services
> such
> > as Amazon AWS, I would think that these problems would be
> > addressable/solvable.
> >
> > regards,
> > CP
> >
> > On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com>
> wrote:
> >
> > > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> cheenpin.lim@gmail.com
> > > >wrote:
> > > >Once you have more load
> > > >applied to the servers/services different characteristics tend to
> > emerge,
> > > >which is why I was thinking that the use of JMeter would be useful.
> > >
> > > *I may totally sound cynical while saying this but honestly the reason
> > why
> > > I am criticising your idea is mainly due to the week rational behind
> your
> > > motivation to create this feature:
> > >
> > > 1. First of all browser induced latency on the total workload is
> grossly
> > > overrated. Having said that, all that an accurate simulation of browser
> > > latency would do to your workload is that it would marginally reduce
> the
> > > total hits made on the server, which I think is bad from a fundamental
> > load
> > > testing perspective.
> > > 2.To accommodate in-memory containers to hold 100s-1000s of virutal
> user
> > > browser sessions is practically impossible without testers having to
> > > significantly size up their test lab resources. Think about the cost
> > > factor.
> > > 3. Looking at todays trend of browser adoption your web driver would
> have
> > > to be equipped with simulation capability multiple types of browsers
> such
> > > as IE, Chromer, Safari etc. etc. and also you would have to consider
> > > simulating various device characteristics on the generated load. Think
> > > about the complexity.
> > >
> > > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > > TrueClient. HP came out with Ajax TrueClient not with an intention
> > > of generating real browser traffic but it was mainly developed to
> reduce
> > > the complexity of scripting apps that employ unintelligible data
> > > serialization like Google GWT. With this HP's protocol you wouldn't
> have
> > a
> > > necessity of correlating session variables at all and there by you
> would
> > be
> > > significantly reducing the time to develop scripts. However, this
> > protocol
> > > currently has poor user adoption due to various limitations.
> > >
> > > Thanks
> > > Chaitanya M Bhatt
> > > http://www.performancecompetence.com
> > >
> > > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > The tools mentioned help a great deal, however they generally look at
> > it
> > > > from a single browser to a single server scenario. Once you have
> more
> > > load
> > > > applied to the servers/services different characteristics tend to
> > emerge,
> > > > which is why I was thinking that the use of JMeter would be useful.
> > > >
> > > > Furthermore, if our primary audience is hitting the site using a
> > browser,
> > > > then would it not make sense to use the same tool to simulate the
> load?
> > > >
> > > > Just my thoughts.
> > > >
> > > > regards,
> > > > CP
> > > >
> > > > On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com>
> > > wrote:
> > > >
> > > > > Pleanty of awesome tools are already available for free to meet
> your
> > > > goals.
> > > > >
> > > > > Free:
> > > > > https://developers.google.com/pagespeed/
> > > > > http://yslow.org/
> > > > >
> > > > > Free or Premium:
> > > > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > > >
> > > > >
> > > > > Selenium has a webdriver which you can plan on extending - if you
> > will.
> > > > > IMHO Jmeter is not the right platform for this.
> > > > > Thanks
> > > > > Chaitnaya M Bhatt
> > > > > http://www.performancecompetence.com
> > > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > > cheenpin.lim@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > After looking at the current HTTP Sampler, I would like to
> propose
> > a
> > > > > > WebDriver based implementation. The problem is that with a lot
> of
> > > > modern
> > > > > > websites using AJAX, advanced CSS3 (transitions/transforms) to do
> > the
> > > > > > rendering, the performance problem may not be 'visible' to the
> > > standard
> > > > > > HTTP Sampler. The idea is that if we could use JMeter to
> control a
> > > > > browser
> > > > > > and collect the time it takes to render each 'page' it would give
> > the
> > > > > users
> > > > > > an understanding of how long a page would take to render.
> > > > > >
> > > > > > Some of the high level goals are as follows:
> > > > > >
> > > > > > 1. Allow any WebDriver supported browser to be used to perform
> > the
> > > > > > Sampling.
> > > > > > 2. Provide a BeanShell/BSF style pane where the user can script
> > the
> > > > > > behaviour of WebDriver.
> > > > > >
> > > > > > I've got a very basic implementation working here:
> > > > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > > > 'webmeter'
> > > > > > branch to see the changes. There are limitations with this proof
> > of
> > > > > > concept, mainly:
> > > > > >
> > > > > > 1. It only creates Firefox instances
> > > > > > 2. I've only provided Javascript lang support using BSF.
> > > > > >
> > > > > > Some outstanding questions:
> > > > > >
> > > > > > 1. How to support Phone (Android/iOS) based WebDriver, as these
> > > seem
> > > > to
> > > > > > expect to only connect to a single phone.
> > > > > >
> > > > > > I would be happy to contribute the changes back to JMeter. Can I
> > get
> > > > > some
> > > > > > feedback on this proposal, and what I need to do to contribute my
> > > > changes
> > > > > > back to the main JMeter codebase?
> > > > > >
> > > > > > regards,
> > > > > > CP Lim
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Shmuel Krakower <sh...@gmail.com>.
I'd suggest to HP to drop C language scripting in order to make scripting
easier / shorter.
As Chaitanya said - browser response times is overrated.
Anyhow, I agree that implementing load tests with hundreds / thousands of
virtual users with that kind of a sampler(which running a browser) will
require much more hardware but as CP said, with today's cloud services this
is very affordable, thus this is not the issue.
I think that there are services on the web which already doing this (
https://browsermob.com/website-load-testing-features), but I never tried
those.
CP, if you get different numbers between you monitoring solution and JMeter
reports, you should try to understand why that happens.
It might be for many reasons. Did developing this sampler resolved your
original problem? Do you get more close numbers now?
Shmuel.
On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <ch...@gmail.com>wrote:
> Hi Chaitanya,
>
> I guess I am looking at this from a browser view rather than from the
> server side. What I tend to find is that current load testing does not
> properly reflect the browser load times on our site. The reports from
> Google (in webmasters tools) and newrelic.com give a very different
> picture
> from that done during load/performance testing.
>
> We use HP Load Runner for our in-house load testing and the numbers we get
> (even when we download 'all' external resources) does not come close to the
> numbers reported by the other tools. I guess I am attempting to address a
> gap that I see in our in-house load/performance testing, and the
> integration of webdriver into JMeter could solve this problem. At this
> stage it is an attempt to try to provide a means to simulate what our
> production servers see (i.e. using a real browser that will spend time
> executing JS/CSS).
>
> Yes, I agree that the complexity and cost of running a browser instance is
> higher than using the standard HTTP Sampler, but with modern services such
> as Amazon AWS, I would think that these problems would be
> addressable/solvable.
>
> regards,
> CP
>
> On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com> wrote:
>
> > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> > >Once you have more load
> > >applied to the servers/services different characteristics tend to
> emerge,
> > >which is why I was thinking that the use of JMeter would be useful.
> >
> > *I may totally sound cynical while saying this but honestly the reason
> why
> > I am criticising your idea is mainly due to the week rational behind your
> > motivation to create this feature:
> >
> > 1. First of all browser induced latency on the total workload is grossly
> > overrated. Having said that, all that an accurate simulation of browser
> > latency would do to your workload is that it would marginally reduce the
> > total hits made on the server, which I think is bad from a fundamental
> load
> > testing perspective.
> > 2.To accommodate in-memory containers to hold 100s-1000s of virutal user
> > browser sessions is practically impossible without testers having to
> > significantly size up their test lab resources. Think about the cost
> > factor.
> > 3. Looking at todays trend of browser adoption your web driver would have
> > to be equipped with simulation capability multiple types of browsers such
> > as IE, Chromer, Safari etc. etc. and also you would have to consider
> > simulating various device characteristics on the generated load. Think
> > about the complexity.
> >
> > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > TrueClient. HP came out with Ajax TrueClient not with an intention
> > of generating real browser traffic but it was mainly developed to reduce
> > the complexity of scripting apps that employ unintelligible data
> > serialization like Google GWT. With this HP's protocol you wouldn't have
> a
> > necessity of correlating session variables at all and there by you would
> be
> > significantly reducing the time to develop scripts. However, this
> protocol
> > currently has poor user adoption due to various limitations.
> >
> > Thanks
> > Chaitanya M Bhatt
> > http://www.performancecompetence.com
> >
> > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > The tools mentioned help a great deal, however they generally look at
> it
> > > from a single browser to a single server scenario. Once you have more
> > load
> > > applied to the servers/services different characteristics tend to
> emerge,
> > > which is why I was thinking that the use of JMeter would be useful.
> > >
> > > Furthermore, if our primary audience is hitting the site using a
> browser,
> > > then would it not make sense to use the same tool to simulate the load?
> > >
> > > Just my thoughts.
> > >
> > > regards,
> > > CP
> > >
> > > On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com>
> > wrote:
> > >
> > > > Pleanty of awesome tools are already available for free to meet your
> > > goals.
> > > >
> > > > Free:
> > > > https://developers.google.com/pagespeed/
> > > > http://yslow.org/
> > > >
> > > > Free or Premium:
> > > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > >
> > > >
> > > > Selenium has a webdriver which you can plan on extending - if you
> will.
> > > > IMHO Jmeter is not the right platform for this.
> > > > Thanks
> > > > Chaitnaya M Bhatt
> > > > http://www.performancecompetence.com
> > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > cheenpin.lim@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > After looking at the current HTTP Sampler, I would like to propose
> a
> > > > > WebDriver based implementation. The problem is that with a lot of
> > > modern
> > > > > websites using AJAX, advanced CSS3 (transitions/transforms) to do
> the
> > > > > rendering, the performance problem may not be 'visible' to the
> > standard
> > > > > HTTP Sampler. The idea is that if we could use JMeter to control a
> > > > browser
> > > > > and collect the time it takes to render each 'page' it would give
> the
> > > > users
> > > > > an understanding of how long a page would take to render.
> > > > >
> > > > > Some of the high level goals are as follows:
> > > > >
> > > > > 1. Allow any WebDriver supported browser to be used to perform
> the
> > > > > Sampling.
> > > > > 2. Provide a BeanShell/BSF style pane where the user can script
> the
> > > > > behaviour of WebDriver.
> > > > >
> > > > > I've got a very basic implementation working here:
> > > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > > 'webmeter'
> > > > > branch to see the changes. There are limitations with this proof
> of
> > > > > concept, mainly:
> > > > >
> > > > > 1. It only creates Firefox instances
> > > > > 2. I've only provided Javascript lang support using BSF.
> > > > >
> > > > > Some outstanding questions:
> > > > >
> > > > > 1. How to support Phone (Android/iOS) based WebDriver, as these
> > seem
> > > to
> > > > > expect to only connect to a single phone.
> > > > >
> > > > > I would be happy to contribute the changes back to JMeter. Can I
> get
> > > > some
> > > > > feedback on this proposal, and what I need to do to contribute my
> > > changes
> > > > > back to the main JMeter codebase?
> > > > >
> > > > > regards,
> > > > > CP Lim
> > > > >
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Cheen-Pin Lim <ch...@gmail.com>.
Hi Chaitanya,
I guess I am looking at this from a browser view rather than from the
server side. What I tend to find is that current load testing does not
properly reflect the browser load times on our site. The reports from
Google (in webmasters tools) and newrelic.com give a very different picture
from that done during load/performance testing.
We use HP Load Runner for our in-house load testing and the numbers we get
(even when we download 'all' external resources) does not come close to the
numbers reported by the other tools. I guess I am attempting to address a
gap that I see in our in-house load/performance testing, and the
integration of webdriver into JMeter could solve this problem. At this
stage it is an attempt to try to provide a means to simulate what our
production servers see (i.e. using a real browser that will spend time
executing JS/CSS).
Yes, I agree that the complexity and cost of running a browser instance is
higher than using the standard HTTP Sampler, but with modern services such
as Amazon AWS, I would think that these problems would be
addressable/solvable.
regards,
CP
On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com> wrote:
> *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
> >Once you have more load
> >applied to the servers/services different characteristics tend to emerge,
> >which is why I was thinking that the use of JMeter would be useful.
>
> *I may totally sound cynical while saying this but honestly the reason why
> I am criticising your idea is mainly due to the week rational behind your
> motivation to create this feature:
>
> 1. First of all browser induced latency on the total workload is grossly
> overrated. Having said that, all that an accurate simulation of browser
> latency would do to your workload is that it would marginally reduce the
> total hits made on the server, which I think is bad from a fundamental load
> testing perspective.
> 2.To accommodate in-memory containers to hold 100s-1000s of virutal user
> browser sessions is practically impossible without testers having to
> significantly size up their test lab resources. Think about the cost
> factor.
> 3. Looking at todays trend of browser adoption your web driver would have
> to be equipped with simulation capability multiple types of browsers such
> as IE, Chromer, Safari etc. etc. and also you would have to consider
> simulating various device characteristics on the generated load. Think
> about the complexity.
>
> @Shmuel: The loadrunner protocol you are talking about is Ajax
> TrueClient. HP came out with Ajax TrueClient not with an intention
> of generating real browser traffic but it was mainly developed to reduce
> the complexity of scripting apps that employ unintelligible data
> serialization like Google GWT. With this HP's protocol you wouldn't have a
> necessity of correlating session variables at all and there by you would be
> significantly reducing the time to develop scripts. However, this protocol
> currently has poor user adoption due to various limitations.
>
> Thanks
> Chaitanya M Bhatt
> http://www.performancecompetence.com
>
> On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi,
> >
> > The tools mentioned help a great deal, however they generally look at it
> > from a single browser to a single server scenario. Once you have more
> load
> > applied to the servers/services different characteristics tend to emerge,
> > which is why I was thinking that the use of JMeter would be useful.
> >
> > Furthermore, if our primary audience is hitting the site using a browser,
> > then would it not make sense to use the same tool to simulate the load?
> >
> > Just my thoughts.
> >
> > regards,
> > CP
> >
> > On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com>
> wrote:
> >
> > > Pleanty of awesome tools are already available for free to meet your
> > goals.
> > >
> > > Free:
> > > https://developers.google.com/pagespeed/
> > > http://yslow.org/
> > >
> > > Free or Premium:
> > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > >
> > >
> > > Selenium has a webdriver which you can plan on extending - if you will.
> > > IMHO Jmeter is not the right platform for this.
> > > Thanks
> > > Chaitnaya M Bhatt
> > > http://www.performancecompetence.com
> > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> cheenpin.lim@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > After looking at the current HTTP Sampler, I would like to propose a
> > > > WebDriver based implementation. The problem is that with a lot of
> > modern
> > > > websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> > > > rendering, the performance problem may not be 'visible' to the
> standard
> > > > HTTP Sampler. The idea is that if we could use JMeter to control a
> > > browser
> > > > and collect the time it takes to render each 'page' it would give the
> > > users
> > > > an understanding of how long a page would take to render.
> > > >
> > > > Some of the high level goals are as follows:
> > > >
> > > > 1. Allow any WebDriver supported browser to be used to perform the
> > > > Sampling.
> > > > 2. Provide a BeanShell/BSF style pane where the user can script the
> > > > behaviour of WebDriver.
> > > >
> > > > I've got a very basic implementation working here:
> > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > 'webmeter'
> > > > branch to see the changes. There are limitations with this proof of
> > > > concept, mainly:
> > > >
> > > > 1. It only creates Firefox instances
> > > > 2. I've only provided Javascript lang support using BSF.
> > > >
> > > > Some outstanding questions:
> > > >
> > > > 1. How to support Phone (Android/iOS) based WebDriver, as these
> seem
> > to
> > > > expect to only connect to a single phone.
> > > >
> > > > I would be happy to contribute the changes back to JMeter. Can I get
> > > some
> > > > feedback on this proposal, and what I need to do to contribute my
> > changes
> > > > back to the main JMeter codebase?
> > > >
> > > > regards,
> > > > CP Lim
> > > >
> > >
> >
>
Re: Web Driver based Sampler?
Posted by chaitanya bhatt <bh...@gmail.com>.
*On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <ch...@gmail.com>wrote:
>Once you have more load
>applied to the servers/services different characteristics tend to emerge,
>which is why I was thinking that the use of JMeter would be useful.
*I may totally sound cynical while saying this but honestly the reason why
I am criticising your idea is mainly due to the week rational behind your
motivation to create this feature:
1. First of all browser induced latency on the total workload is grossly
overrated. Having said that, all that an accurate simulation of browser
latency would do to your workload is that it would marginally reduce the
total hits made on the server, which I think is bad from a fundamental load
testing perspective.
2.To accommodate in-memory containers to hold 100s-1000s of virutal user
browser sessions is practically impossible without testers having to
significantly size up their test lab resources. Think about the cost factor.
3. Looking at todays trend of browser adoption your web driver would have
to be equipped with simulation capability multiple types of browsers such
as IE, Chromer, Safari etc. etc. and also you would have to consider
simulating various device characteristics on the generated load. Think
about the complexity.
@Shmuel: The loadrunner protocol you are talking about is Ajax
TrueClient. HP came out with Ajax TrueClient not with an intention
of generating real browser traffic but it was mainly developed to reduce
the complexity of scripting apps that employ unintelligible data
serialization like Google GWT. With this HP's protocol you wouldn't have a
necessity of correlating session variables at all and there by you would be
significantly reducing the time to develop scripts. However, this protocol
currently has poor user adoption due to various limitations.
Thanks
Chaitanya M Bhatt
http://www.performancecompetence.com
On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <ch...@gmail.com>wrote:
> Hi,
>
> The tools mentioned help a great deal, however they generally look at it
> from a single browser to a single server scenario. Once you have more load
> applied to the servers/services different characteristics tend to emerge,
> which is why I was thinking that the use of JMeter would be useful.
>
> Furthermore, if our primary audience is hitting the site using a browser,
> then would it not make sense to use the same tool to simulate the load?
>
> Just my thoughts.
>
> regards,
> CP
>
> On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com> wrote:
>
> > Pleanty of awesome tools are already available for free to meet your
> goals.
> >
> > Free:
> > https://developers.google.com/pagespeed/
> > http://yslow.org/
> >
> > Free or Premium:
> > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> >
> >
> > Selenium has a webdriver which you can plan on extending - if you will.
> > IMHO Jmeter is not the right platform for this.
> > Thanks
> > Chaitnaya M Bhatt
> > http://www.performancecompetence.com
> > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > After looking at the current HTTP Sampler, I would like to propose a
> > > WebDriver based implementation. The problem is that with a lot of
> modern
> > > websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> > > rendering, the performance problem may not be 'visible' to the standard
> > > HTTP Sampler. The idea is that if we could use JMeter to control a
> > browser
> > > and collect the time it takes to render each 'page' it would give the
> > users
> > > an understanding of how long a page would take to render.
> > >
> > > Some of the high level goals are as follows:
> > >
> > > 1. Allow any WebDriver supported browser to be used to perform the
> > > Sampling.
> > > 2. Provide a BeanShell/BSF style pane where the user can script the
> > > behaviour of WebDriver.
> > >
> > > I've got a very basic implementation working here:
> > > https://github.com/cplim/JMeter - you'll have to switch to the
> > 'webmeter'
> > > branch to see the changes. There are limitations with this proof of
> > > concept, mainly:
> > >
> > > 1. It only creates Firefox instances
> > > 2. I've only provided Javascript lang support using BSF.
> > >
> > > Some outstanding questions:
> > >
> > > 1. How to support Phone (Android/iOS) based WebDriver, as these seem
> to
> > > expect to only connect to a single phone.
> > >
> > > I would be happy to contribute the changes back to JMeter. Can I get
> > some
> > > feedback on this proposal, and what I need to do to contribute my
> changes
> > > back to the main JMeter codebase?
> > >
> > > regards,
> > > CP Lim
> > >
> >
>
Re: Web Driver based Sampler?
Posted by Cheen-Pin Lim <ch...@gmail.com>.
Hi,
The tools mentioned help a great deal, however they generally look at it
from a single browser to a single server scenario. Once you have more load
applied to the servers/services different characteristics tend to emerge,
which is why I was thinking that the use of JMeter would be useful.
Furthermore, if our primary audience is hitting the site using a browser,
then would it not make sense to use the same tool to simulate the load?
Just my thoughts.
regards,
CP
On 21 April 2012 17:26, chaitanya bhatt <bh...@gmail.com> wrote:
> Pleanty of awesome tools are already available for free to meet your goals.
>
> Free:
> https://developers.google.com/pagespeed/
> http://yslow.org/
>
> Free or Premium:
> http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
>
>
> Selenium has a webdriver which you can plan on extending - if you will.
> IMHO Jmeter is not the right platform for this.
> Thanks
> Chaitnaya M Bhatt
> http://www.performancecompetence.com
> On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi,
> >
> > After looking at the current HTTP Sampler, I would like to propose a
> > WebDriver based implementation. The problem is that with a lot of modern
> > websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> > rendering, the performance problem may not be 'visible' to the standard
> > HTTP Sampler. The idea is that if we could use JMeter to control a
> browser
> > and collect the time it takes to render each 'page' it would give the
> users
> > an understanding of how long a page would take to render.
> >
> > Some of the high level goals are as follows:
> >
> > 1. Allow any WebDriver supported browser to be used to perform the
> > Sampling.
> > 2. Provide a BeanShell/BSF style pane where the user can script the
> > behaviour of WebDriver.
> >
> > I've got a very basic implementation working here:
> > https://github.com/cplim/JMeter - you'll have to switch to the
> 'webmeter'
> > branch to see the changes. There are limitations with this proof of
> > concept, mainly:
> >
> > 1. It only creates Firefox instances
> > 2. I've only provided Javascript lang support using BSF.
> >
> > Some outstanding questions:
> >
> > 1. How to support Phone (Android/iOS) based WebDriver, as these seem to
> > expect to only connect to a single phone.
> >
> > I would be happy to contribute the changes back to JMeter. Can I get
> some
> > feedback on this proposal, and what I need to do to contribute my changes
> > back to the main JMeter codebase?
> >
> > regards,
> > CP Lim
> >
>
Re: Web Driver based Sampler?
Posted by chaitanya bhatt <bh...@gmail.com>.
Pleanty of awesome tools are already available for free to meet your goals.
Free:
https://developers.google.com/pagespeed/
http://yslow.org/
Free or Premium:
http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
Selenium has a webdriver which you can plan on extending - if you will.
IMHO Jmeter is not the right platform for this.
Thanks
Chaitnaya M Bhatt
http://www.performancecompetence.com
On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <ch...@gmail.com>wrote:
> Hi,
>
> After looking at the current HTTP Sampler, I would like to propose a
> WebDriver based implementation. The problem is that with a lot of modern
> websites using AJAX, advanced CSS3 (transitions/transforms) to do the
> rendering, the performance problem may not be 'visible' to the standard
> HTTP Sampler. The idea is that if we could use JMeter to control a browser
> and collect the time it takes to render each 'page' it would give the users
> an understanding of how long a page would take to render.
>
> Some of the high level goals are as follows:
>
> 1. Allow any WebDriver supported browser to be used to perform the
> Sampling.
> 2. Provide a BeanShell/BSF style pane where the user can script the
> behaviour of WebDriver.
>
> I've got a very basic implementation working here:
> https://github.com/cplim/JMeter - you'll have to switch to the 'webmeter'
> branch to see the changes. There are limitations with this proof of
> concept, mainly:
>
> 1. It only creates Firefox instances
> 2. I've only provided Javascript lang support using BSF.
>
> Some outstanding questions:
>
> 1. How to support Phone (Android/iOS) based WebDriver, as these seem to
> expect to only connect to a single phone.
>
> I would be happy to contribute the changes back to JMeter. Can I get some
> feedback on this proposal, and what I need to do to contribute my changes
> back to the main JMeter codebase?
>
> regards,
> CP Lim
>