You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@jmeter.apache.org by "Lloyd, Oliver" <ol...@newsint.co.uk> on 2011/04/30 03:23:27 UTC

How to handle JMeter/CI results stored in a database?

Hello JMeter people,

Can I ask for your valuable opinions? I'm looking for
some recommendations on what software / process to use to present multiple
sets of test results stored in a database. I want a simple webpage view (or
even a snazzy dashboard) that shows the trend from build to build for
various KPIs. I've basically setup a CI system and now want to spend more
time at home. Any ideas?

I'm NOT looking to directly parse JTL files and produce graphs analysing a
single test run, that's pretty straight forward and I'm actually happier
with this as a hands-on process - you can't automate thinking!

But we are churning out code at an amazing rate these days and we are
looking for ways to do this even faster, and even better, and having a great
tool that we can use to effectively and automatically handle results is key
to this. Sure, we could code something up (and may yet chose this option)
but I'm interested in what's out there.

By the way, I have already looked into this a lot. Some of the CI plugins
look great but they are often too simple and the big BI offerings are
probably overkill. In the middle is this mountain of reporting tools -
everyone I talk to has their favourite - and it is for this reason I am
asking for the opinion of the people with the actual battle scars.

Thanks for you time.
Oliver
P.s. If anyone has done this with Google Gadgets directly calling the data
source I'd be really interested to hear about it.

--
"Please consider the environment before printing this e-mail"

The Newspaper Marketing Agency: Opening Up Newspapers:
www.nmauk.co.uk

This e-mail and any attachments are confidential, may be legally privileged and are the property of
News International Limited (which is the holding company for the News International group, is
registered in England under number 81701 and whose registered office is 3 Thomas More Square,
London E98 1XY, VAT number GB 243 8054 69), on whose systems they were generated.

If you have received this e-mail in error, please notify the sender immediately and do not use,
distribute, store or copy it in any way. Statements or opinions in this e-mail or any attachment are
those of the author and are not necessarily agreed or authorised by News International Limited or
any member of its group. News International Limited may monitor outgoing or incoming emails as
permitted by law. It accepts no liability for viruses introduced by this e-mail or attachments.

Re: How to handle JMeter/CI results stored in a database?

Posted by oliver <ol...@hotmail.com>.

In the end I opted for the custom path. Time consuming but fun.

Along the way though, I also noticed a new service that some of you might be
interested in:  http://www.geckoboard.com Geckoboard . It's only been live a
few months but it offers a really nice dashboard front-end to CI results. If
you have a lot of datasets from multiple tests run overnight then it's a
simple way to collate all the key metrics in one place.

I don't think it's really aimed at this but with a bit of work I set it up
so I get the standard results from the last test (throughput, tps, 90th
percentiles over time, etc.) - one page for each project. But I also have a
view giving trends over the past few weeks and I can overlay the results
from the latest test with those from the previous one. 

None of this replaces actually analysing data, actually *doing testing*, but
for automated CI it's the d's bs. Oh, and, if I had an iPad or iPhone, I
could view it all there too.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-to-handle-JMeter-CI-results-stored-in-a-database-tp4360310p4396606.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: How to handle JMeter/CI results stored in a database?

Posted by "Lloyd, Oliver" <ol...@newsint.co.uk>.

I figured that you had some sort of custom reporting. And you're right, it
is very much an option that we will create our own reporting layer - there
are loads of talented people sat around me who could do this in their sleep
- it may even work well as a POC for some of the new technologies we are
looking at.

But before starting off on that road I am interested in what third-party
reporting solutions are available.

In general, this has always been a gaping hole in performance testing, every
major tool I have used lacks this ability to report effectively over
multiple test runs - lots of sexy deep analysis of that test I just ran, but
try trending over three, four or just two runs and things get really messy.
This was the main reason we moved open-source for CI, we don't need deep
diagnostics and automatically generated word reports, we need key
benchmarking results dynamically published on a webpage, viewed on iPads and
shown on a big screen at the end of the office for everyone to see. Or
something like that.

--
"Please consider the environment before printing this e-mail"

The Newspaper Marketing Agency: Opening Up Newspapers:
www.nmauk.co.uk

Re: How to handle JMeter/CI results stored in a database?

Posted by Bruce Ide <fl...@gmail.com>.

Our Hudson guy wrote a XSLT script to transform the jmeter report into an
HTML report, which Hudson displays as a test artifact. I don't know if you
can show that stuff on Hudson's opening page though -- we drill down from
the page with all the tests to look at the results of the tests we're
interested in. His isn't terribly fancy either, but once you're processing
the XML with something, you have a lot of options for displaying the result.

If you've already got the data you want to display and have an idea of how
you want it displayed, I wonder if one of the web service frameworks like
Ruby on Rails might work better for you? Those frameworks seem to be pretty
good at letting you build a fast application to slice and dice your data
however you want to. You could probably build some graphs with gnuplot
through a framework like that, too. That might be more work than you want to
put into it, though.

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: How to handle JMeter/CI results stored in a database?

Posted by "Lloyd, Oliver" <ol...@newsint.co.uk>.

Hi Bruce,

I looked at Hudson and the integration it offers is great (I had to write
shell scripts and use property files to get this data past through) but from
what I could see it only seemed to give one summary response time graph and
a report on the error rate - can you change that? I need to be able to break
things down by individual request types (Login, Search etc) and I also have
a lot of data from system monitoring, which is often custom, app-specific
stuff.

Thing is, I already have the detailed, aggregated and summarised data
sitting patiently in mySQL - I just need that final piece of the
framework...

On 30 April 2011 02:55, Bruce Ide <fl...@gmail.com> wrote:

> My team is using hudson for that sort of thing and it seems to be pretty
> nice. We're calling it with an ant task and I have a BSF sampler kludge for
> getting parameters into Jmeter from Hudson. It's pretty easy to kick off a
> big batch of tests and distribute them across multiple machines, and it
> does
> a good job of summarizing the results and the trend for the last few runs.
>

-- 
"Please consider the environment before printing this e-mail" 

The Newspaper Marketing Agency: Opening Up Newspapers: 
www.nmauk.co.uk 

This e-mail and any attachments are confidential, may be legally privileged and are the property of 
News International Limited (which is the holding company for the News International group, is 
registered in England under number 81701 and whose registered office is 3 Thomas More Square, 
London E98 1XY, VAT number GB 243 8054 69), on whose systems they were generated. 

If you have received this e-mail in error, please notify the sender immediately and do not use, 
distribute, store or copy it in any way. Statements or opinions in this e-mail or any attachment are 
those of the author and are not necessarily agreed or authorised by News International Limited or  
any member of its group. News International Limited may monitor outgoing or incoming emails as 
permitted by law. It accepts no liability for viruses introduced by this e-mail or attachments.

Re: How to handle JMeter/CI results stored in a database?

Posted by Bruce Ide <fl...@gmail.com>.

My team is using hudson for that sort of thing and it seems to be pretty
nice. We're calling it with an ant task and I have a BSF sampler kludge for
getting parameters into Jmeter from Hudson. It's pretty easy to kick off a
big batch of tests and distribute them across multiple machines, and it does
a good job of summarizing the results and the trend for the last few runs.

-- 
Bruce Ide
FlyingRhenquest@gmail.com