Re: Web Driver based Sampler?

[Shmuel Krakower <sh...@gmail.com>]

Hi CP,
After looking at the sampler - one question in mind:
I might want to get more details on specific action inside the script, i.e:
browse into google.com
search for a value

I'd like to know how much time the search took - and I can't find an easy
solution for this.
Any ideas?

Regards,
Shmuel.

On Mon, Apr 23, 2012 at 3:38 AM, Cheen-Pin Lim <ch...@gmail.com>wrote:

> Hi Shmuel,
>
> I've yet to try the WebSampler implementation on our test environment.
>  Unfortunately I'm away for the next 2 weeks, but when I get back to work I
> can try out the WebSampler and see if I can replicate the statistics I'm
> getting in our production environment.
>
> regards,
> CP
>
> On 22 April 2012 23:49, Shmuel Krakower <sh...@gmail.com> wrote:
>
> > I'd suggest to HP to drop C language scripting in order to make scripting
> > easier / shorter.
> >
> > As Chaitanya said - browser response times is overrated.
> >
> > Anyhow, I agree that implementing load tests with hundreds / thousands of
> > virtual users with that kind of a sampler(which running a browser) will
> > require much more hardware but as CP said, with today's cloud services
> this
> > is very affordable, thus this is not the issue.
> > I think that there are services on the web which already doing this (
> > https://browsermob.com/website-load-testing-features), but I never tried
> > those.
> >
> > CP, if you get different numbers between you monitoring solution and
> JMeter
> > reports, you should try to understand why that happens.
> > It might be for many reasons. Did developing this sampler resolved your
> > original problem? Do you get more close numbers now?
> >
> > Shmuel.
> >
> >
> > On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > >wrote:
> >
> > > Hi Chaitanya,
> > >
> > > I guess I am looking at this from a browser view rather than from the
> > > server side.  What I tend to find is that current load testing does not
> > > properly reflect the browser load times on our site.  The reports from
> > > Google (in webmasters tools) and newrelic.com give a very different
> > > picture
> > > from that done during load/performance testing.
> > >
> > > We use HP Load Runner for our in-house load testing and the numbers we
> > get
> > > (even when we download 'all' external resources) does not come close to
> > the
> > > numbers reported by the other tools.  I guess I am attempting to
> address
> > a
> > > gap that I see in our in-house load/performance testing, and the
> > > integration of webdriver into JMeter could solve this problem.  At this
> > > stage it is an attempt to try to provide a means to simulate what our
> > > production servers see (i.e. using a real browser that will spend time
> > > executing JS/CSS).
> > >
> > > Yes, I agree that the complexity and cost of running a browser instance
> > is
> > > higher than using the standard HTTP Sampler, but with modern services
> > such
> > > as Amazon AWS, I would think that these problems would be
> > > addressable/solvable.
> > >
> > > regards,
> > > CP
> > >
> > > On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com>
> > wrote:
> > >
> > > > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > cheenpin.lim@gmail.com
> > > > >wrote:
> > > > >Once you have more load
> > > > >applied to the servers/services different characteristics tend to
> > > emerge,
> > > > >which is why I was thinking that the use of JMeter would be useful.
> > > >
> > > > *I may totally sound cynical while saying this but honestly the
> reason
> > > why
> > > > I am criticising your idea is mainly due to the week rational behind
> > your
> > > > motivation to create this feature:
> > > >
> > > > 1. First of all browser induced latency on the total workload is
> > grossly
> > > > overrated. Having said that, all that an accurate simulation of
> browser
> > > > latency would do to your workload is that it would marginally reduce
> > the
> > > > total hits made on the server, which I think is bad from a
> fundamental
> > > load
> > > > testing perspective.
> > > > 2.To accommodate in-memory containers to hold 100s-1000s of virutal
> > user
> > > > browser sessions is practically impossible without testers having to
> > > > significantly size up their test lab resources. Think about the cost
> > > > factor.
> > > > 3. Looking at todays trend of browser adoption your web driver would
> > have
> > > > to be equipped with simulation capability multiple types of browsers
> > such
> > > > as IE, Chromer, Safari etc. etc. and also you would have to consider
> > > > simulating various device characteristics on the generated load.
> Think
> > > > about the complexity.
> > > >
> > > > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > > > TrueClient. HP came out with Ajax TrueClient not with an intention
> > > > of generating real browser traffic but it was mainly developed to
> > reduce
> > > > the complexity of scripting apps that employ unintelligible data
> > > > serialization like Google GWT. With this HP's protocol you wouldn't
> > have
> > > a
> > > > necessity of correlating session variables at all and there by you
> > would
> > > be
> > > > significantly reducing the time to develop scripts. However, this
> > > protocol
> > > > currently has poor user adoption due to various limitations.
> > > >
> > > > Thanks
> > > > Chaitanya M Bhatt
> > > > http://www.performancecompetence.com
> > > >
> > > > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> cheenpin.lim@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > The tools mentioned help a great deal, however they generally look
> at
> > > it
> > > > > from a single browser to a single server scenario.  Once you have
> > more
> > > > load
> > > > > applied to the servers/services different characteristics tend to
> > > emerge,
> > > > > which is why I was thinking that the use of JMeter would be useful.
> > > > >
> > > > > Furthermore, if our primary audience is hitting the site using a
> > > browser,
> > > > > then would it not make sense to use the same tool to simulate the
> > load?
> > > > >
> > > > > Just my thoughts.
> > > > >
> > > > > regards,
> > > > > CP
> > > > >
> > > > > On 21 April 2012 17:26, chaitanya bhatt <bhatt.chaitanya@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Pleanty of awesome tools are already available for free to meet
> > your
> > > > > goals.
> > > > > >
> > > > > > Free:
> > > > > > https://developers.google.com/pagespeed/
> > > > > > http://yslow.org/
> > > > > >
> > > > > > Free or Premium:
> > > > > > http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > > > >
> > > > > >
> > > > > > Selenium has a webdriver which you can plan on extending - if you
> > > will.
> > > > > > IMHO Jmeter is not the right platform for this.
> > > > > > Thanks
> > > > > > Chaitnaya M Bhatt
> > > > > > http://www.performancecompetence.com
> > > > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > > > cheenpin.lim@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > After looking at the current HTTP Sampler, I would like to
> > propose
> > > a
> > > > > > > WebDriver based implementation.  The problem is that with a lot
> > of
> > > > > modern
> > > > > > > websites using AJAX, advanced CSS3 (transitions/transforms) to
> do
> > > the
> > > > > > > rendering, the performance problem may not be 'visible' to the
> > > > standard
> > > > > > > HTTP Sampler.  The idea is that if we could use JMeter to
> > control a
> > > > > > browser
> > > > > > > and collect the time it takes to render each 'page' it would
> give
> > > the
> > > > > > users
> > > > > > > an understanding of how long a page would take to render.
> > > > > > >
> > > > > > > Some of the high level goals are as follows:
> > > > > > >
> > > > > > >   1. Allow any WebDriver supported browser to be used to
> perform
> > > the
> > > > > > >   Sampling.
> > > > > > >   2. Provide a BeanShell/BSF style pane where the user can
> script
> > > the
> > > > > > >   behaviour of WebDriver.
> > > > > > >
> > > > > > > I've got a very basic implementation working here:
> > > > > > > https://github.com/cplim/JMeter - you'll have to switch to the
> > > > > > 'webmeter'
> > > > > > > branch to see the changes.  There are limitations with this
> proof
> > > of
> > > > > > > concept, mainly:
> > > > > > >
> > > > > > >   1. It only creates Firefox instances
> > > > > > >   2. I've only provided Javascript lang support using BSF.
> > > > > > >
> > > > > > > Some outstanding questions:
> > > > > > >
> > > > > > >   1. How to support Phone (Android/iOS) based WebDriver, as
> these
> > > > seem
> > > > > to
> > > > > > >   expect to only connect to a single phone.
> > > > > > >
> > > > > > > I would be happy to contribute the changes back to JMeter.
>  Can I
> > > get
> > > > > > some
> > > > > > > feedback on this proposal, and what I need to do to contribute
> my
> > > > > changes
> > > > > > > back to the main JMeter codebase?
> > > > > > >
> > > > > > > regards,
> > > > > > > CP Lim
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Web Driver based Sampler?

[Cheen-Pin Lim <ch...@gmail.com>]

Hi Shmuel,

My initial thoughts on this is to perform this in separate samplers.  If
the user wishes to do the following:

1. Go to Google.com
2. Perform a search for cheese
3. Click on the first result.

I would represent this with 3 Samplers for each of the steps above.  This
way we can track how long each of the pages (home page, search results,
first search result) by adding a Listener to each of the samplers.

This way I accomplished this is by re-using the same WebDriver (browser)
instance between each of the Samplers, and storing this in a ThreadlLocal
variable.

Hope this answers your question.  If not, I can send you a JMX file that
describes this behaviour (can send it to your gmail account instead of
spamming this list).

regards,
CP

On 7 May 2012 22:30, Shmuel Krakower <sh...@gmail.com> wrote:

> Hi CP,
> After looking at the sampler - one question in mind:
> I might want to get more details on specific action inside the script, i.e:
> browse into google.com
> search for a value
>
> I'd like to know how much time the search took - and I can't find an easy
> solution for this.
> Any ideas?
>
> Regards,
> Shmuel.
>
> On Mon, Apr 23, 2012 at 3:38 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> >wrote:
>
> > Hi Shmuel,
> >
> > I've yet to try the WebSampler implementation on our test environment.
> >  Unfortunately I'm away for the next 2 weeks, but when I get back to
> work I
> > can try out the WebSampler and see if I can replicate the statistics I'm
> > getting in our production environment.
> >
> > regards,
> > CP
> >
> > On 22 April 2012 23:49, Shmuel Krakower <sh...@gmail.com> wrote:
> >
> > > I'd suggest to HP to drop C language scripting in order to make
> scripting
> > > easier / shorter.
> > >
> > > As Chaitanya said - browser response times is overrated.
> > >
> > > Anyhow, I agree that implementing load tests with hundreds / thousands
> of
> > > virtual users with that kind of a sampler(which running a browser) will
> > > require much more hardware but as CP said, with today's cloud services
> > this
> > > is very affordable, thus this is not the issue.
> > > I think that there are services on the web which already doing this (
> > > https://browsermob.com/website-load-testing-features), but I never
> tried
> > > those.
> > >
> > > CP, if you get different numbers between you monitoring solution and
> > JMeter
> > > reports, you should try to understand why that happens.
> > > It might be for many reasons. Did developing this sampler resolved your
> > > original problem? Do you get more close numbers now?
> > >
> > > Shmuel.
> > >
> > >
> > > On Sun, Apr 22, 2012 at 5:27 AM, Cheen-Pin Lim <cheenpin.lim@gmail.com
> > > >wrote:
> > >
> > > > Hi Chaitanya,
> > > >
> > > > I guess I am looking at this from a browser view rather than from the
> > > > server side.  What I tend to find is that current load testing does
> not
> > > > properly reflect the browser load times on our site.  The reports
> from
> > > > Google (in webmasters tools) and newrelic.com give a very different
> > > > picture
> > > > from that done during load/performance testing.
> > > >
> > > > We use HP Load Runner for our in-house load testing and the numbers
> we
> > > get
> > > > (even when we download 'all' external resources) does not come close
> to
> > > the
> > > > numbers reported by the other tools.  I guess I am attempting to
> > address
> > > a
> > > > gap that I see in our in-house load/performance testing, and the
> > > > integration of webdriver into JMeter could solve this problem.  At
> this
> > > > stage it is an attempt to try to provide a means to simulate what our
> > > > production servers see (i.e. using a real browser that will spend
> time
> > > > executing JS/CSS).
> > > >
> > > > Yes, I agree that the complexity and cost of running a browser
> instance
> > > is
> > > > higher than using the standard HTTP Sampler, but with modern services
> > > such
> > > > as Amazon AWS, I would think that these problems would be
> > > > addressable/solvable.
> > > >
> > > > regards,
> > > > CP
> > > >
> > > > On 22 April 2012 06:30, chaitanya bhatt <bh...@gmail.com>
> > > wrote:
> > > >
> > > > > *On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > > cheenpin.lim@gmail.com
> > > > > >wrote:
> > > > > >Once you have more load
> > > > > >applied to the servers/services different characteristics tend to
> > > > emerge,
> > > > > >which is why I was thinking that the use of JMeter would be
> useful.
> > > > >
> > > > > *I may totally sound cynical while saying this but honestly the
> > reason
> > > > why
> > > > > I am criticising your idea is mainly due to the week rational
> behind
> > > your
> > > > > motivation to create this feature:
> > > > >
> > > > > 1. First of all browser induced latency on the total workload is
> > > grossly
> > > > > overrated. Having said that, all that an accurate simulation of
> > browser
> > > > > latency would do to your workload is that it would marginally
> reduce
> > > the
> > > > > total hits made on the server, which I think is bad from a
> > fundamental
> > > > load
> > > > > testing perspective.
> > > > > 2.To accommodate in-memory containers to hold 100s-1000s of virutal
> > > user
> > > > > browser sessions is practically impossible without testers having
> to
> > > > > significantly size up their test lab resources. Think about the
> cost
> > > > > factor.
> > > > > 3. Looking at todays trend of browser adoption your web driver
> would
> > > have
> > > > > to be equipped with simulation capability multiple types of
> browsers
> > > such
> > > > > as IE, Chromer, Safari etc. etc. and also you would have to
> consider
> > > > > simulating various device characteristics on the generated load.
> > Think
> > > > > about the complexity.
> > > > >
> > > > > @Shmuel: The loadrunner protocol you are talking about is Ajax
> > > > > TrueClient. HP came out with Ajax TrueClient not with an intention
> > > > > of generating real browser traffic but it was mainly developed to
> > > reduce
> > > > > the complexity of scripting apps that employ unintelligible data
> > > > > serialization like Google GWT. With this HP's protocol you wouldn't
> > > have
> > > > a
> > > > > necessity of correlating session variables at all and there by you
> > > would
> > > > be
> > > > > significantly reducing the time to develop scripts. However, this
> > > > protocol
> > > > > currently has poor user adoption due to various limitations.
> > > > >
> > > > > Thanks
> > > > > Chaitanya M Bhatt
> > > > > http://www.performancecompetence.com
> > > > >
> > > > > On Sat, Apr 21, 2012 at 4:07 AM, Cheen-Pin Lim <
> > cheenpin.lim@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > The tools mentioned help a great deal, however they generally
> look
> > at
> > > > it
> > > > > > from a single browser to a single server scenario.  Once you have
> > > more
> > > > > load
> > > > > > applied to the servers/services different characteristics tend to
> > > > emerge,
> > > > > > which is why I was thinking that the use of JMeter would be
> useful.
> > > > > >
> > > > > > Furthermore, if our primary audience is hitting the site using a
> > > > browser,
> > > > > > then would it not make sense to use the same tool to simulate the
> > > load?
> > > > > >
> > > > > > Just my thoughts.
> > > > > >
> > > > > > regards,
> > > > > > CP
> > > > > >
> > > > > > On 21 April 2012 17:26, chaitanya bhatt <
> bhatt.chaitanya@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Pleanty of awesome tools are already available for free to meet
> > > your
> > > > > > goals.
> > > > > > >
> > > > > > > Free:
> > > > > > > https://developers.google.com/pagespeed/
> > > > > > > http://yslow.org/
> > > > > > >
> > > > > > > Free or Premium:
> > > > > > >
> http://ajax.dynatrace.com/ajax/en/content/c-speed-page-load.aspx
> > > > > > >
> > > > > > >
> > > > > > > Selenium has a webdriver which you can plan on extending - if
> you
> > > > will.
> > > > > > > IMHO Jmeter is not the right platform for this.
> > > > > > > Thanks
> > > > > > > Chaitnaya M Bhatt
> > > > > > > http://www.performancecompetence.com
> > > > > > > On Fri, Apr 20, 2012 at 11:55 PM, Cheen-Pin Lim <
> > > > > cheenpin.lim@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > After looking at the current HTTP Sampler, I would like to
> > > propose
> > > > a
> > > > > > > > WebDriver based implementation.  The problem is that with a
> lot
> > > of
> > > > > > modern
> > > > > > > > websites using AJAX, advanced CSS3 (transitions/transforms)
> to
> > do
> > > > the
> > > > > > > > rendering, the performance problem may not be 'visible' to
> the
> > > > > standard
> > > > > > > > HTTP Sampler.  The idea is that if we could use JMeter to
> > > control a
> > > > > > > browser
> > > > > > > > and collect the time it takes to render each 'page' it would
> > give
> > > > the
> > > > > > > users
> > > > > > > > an understanding of how long a page would take to render.
> > > > > > > >
> > > > > > > > Some of the high level goals are as follows:
> > > > > > > >
> > > > > > > >   1. Allow any WebDriver supported browser to be used to
> > perform
> > > > the
> > > > > > > >   Sampling.
> > > > > > > >   2. Provide a BeanShell/BSF style pane where the user can
> > script
> > > > the
> > > > > > > >   behaviour of WebDriver.
> > > > > > > >
> > > > > > > > I've got a very basic implementation working here:
> > > > > > > > https://github.com/cplim/JMeter - you'll have to switch to
> the
> > > > > > > 'webmeter'
> > > > > > > > branch to see the changes.  There are limitations with this
> > proof
> > > > of
> > > > > > > > concept, mainly:
> > > > > > > >
> > > > > > > >   1. It only creates Firefox instances
> > > > > > > >   2. I've only provided Javascript lang support using BSF.
> > > > > > > >
> > > > > > > > Some outstanding questions:
> > > > > > > >
> > > > > > > >   1. How to support Phone (Android/iOS) based WebDriver, as
> > these
> > > > > seem
> > > > > > to
> > > > > > > >   expect to only connect to a single phone.
> > > > > > > >
> > > > > > > > I would be happy to contribute the changes back to JMeter.
> >  Can I
> > > > get
> > > > > > > some
> > > > > > > > feedback on this proposal, and what I need to do to
> contribute
> > my
> > > > > > changes
> > > > > > > > back to the main JMeter codebase?
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > CP Lim
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>