You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Mark Miller <ma...@gmail.com> on 2013/02/07 20:09:51 UTC
Exit code as indicator of (functional) test outcome
I am in the process of integrating *discreet, functional unit
tests*(leveraging several JMeter capabilities) with an enterprise
monitoring
system (Orion). The monitoring system can integrate with any executable
based on exit status.
I've done some searching and read other threads from folks seemingly
searching for a similar capability.
I've come up with and successfully tested the following solution and wanted
to float it to the community to ask for feedback or other suggestions:
In the tesplan, I include a Generate Summary Results listener.
I wrap the execution of JMeter in a simple shell script used to evaluate
the string put to stdout by the listener:
#!/bin/bash
if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx | grep "0
(0.00%)$"
then
echo Success
exit 0
else
echo Failure
exit 1
fi
I have noted that, in a long running test, Generate Summary Results will
output n+ lines over time. But it seems safe that the final line output
terminates ($) with the final overall error rate.
Anyone have any other cautions / considerations / approaches to this?
Thank you.
Mark
Re: Exit code as indicator of (functional) test outcome
Posted by sebb <se...@gmail.com>.
On 7 February 2013 23:40, Mark Miller <ma...@gmail.com> wrote:
> Thanks Shmuel. I did find that suggestion that you'd made on that thread
> and I experimented with it. It did work. However, I wondered that it might
> be overly 'abrupt' in terms of JMeter exiting and finishing writing other
> listeners (file handling), etc. I have not taken the time to look into
> this, though. The strength of that approach is that it is more portable.
>
> Do you happen to know how JMeter handles it's own termination under a
> System.exit via BeanShell?
Same as for any other Java application.
AFAIK there's no way to catch System.exit - and anyway JMeter does not try.
> Mark
>
>
> On Thu, Feb 7, 2013 at 4:15 PM, Shmuel Krakower <sh...@gmail.com> wrote:
>
>>
>> http://jmeter.512774.n5.nabble.com/exiting-with-an-error-return-code-when-Assertion-Test-fails-td5715096.html
>>
>> ("....you can use the "System.exit(ERROR_CODE);" inside a beanshell
>> sampler.")
>>
>> Shmuel Krakower.
>> www.Beatsoo.org - re-use your jmeter scripts for application performance
>> monitoring from worldwide locations for free.
>>
>>
>> On Thu, Feb 7, 2013 at 9:09 PM, Mark Miller <ma...@gmail.com>
>> wrote:
>>
>> > I am in the process of integrating *discreet, functional unit
>> > tests*(leveraging several JMeter capabilities) with an enterprise
>> > monitoring
>> > system (Orion). The monitoring system can integrate with any executable
>> > based on exit status.
>> >
>> > I've done some searching and read other threads from folks seemingly
>> > searching for a similar capability.
>> >
>> > I've come up with and successfully tested the following solution and
>> wanted
>> > to float it to the community to ask for feedback or other suggestions:
>> >
>> > In the tesplan, I include a Generate Summary Results listener.
>> > I wrap the execution of JMeter in a simple shell script used to evaluate
>> > the string put to stdout by the listener:
>> >
>> > #!/bin/bash
>> > if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx | grep
>> "0
>> > (0.00%)$"
>> > then
>> > echo Success
>> > exit 0
>> > else
>> > echo Failure
>> > exit 1
>> > fi
>> >
>> > I have noted that, in a long running test, Generate Summary Results will
>> > output n+ lines over time. But it seems safe that the final line output
>> > terminates ($) with the final overall error rate.
>> >
>> > Anyone have any other cautions / considerations / approaches to this?
>> >
>> > Thank you.
>> >
>> > Mark
>> >
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Exit code as indicator of (functional) test outcome
Posted by Shmuel Krakower <sh...@gmail.com>.
Good question, I have no idea but maybe one of the developers will tell.
The only thing I know is that in the service below(in my signature) I tend
to KILL the JVM (externally) and I didn't notice any open files left open
or any kind of "leaks"... but you are right, it might not be a clean
solution.
Best,
Shmuel Krakower.
www.Beatsoo.org - re-use your jmeter scripts for application performance
monitoring from worldwide locations for free.
On Fri, Feb 8, 2013 at 1:40 AM, Mark Miller <ma...@gmail.com> wrote:
> Thanks Shmuel. I did find that suggestion that you'd made on that thread
> and I experimented with it. It did work. However, I wondered that it might
> be overly 'abrupt' in terms of JMeter exiting and finishing writing other
> listeners (file handling), etc. I have not taken the time to look into
> this, though. The strength of that approach is that it is more portable.
>
> Do you happen to know how JMeter handles it's own termination under a
> System.exit via BeanShell?
>
> Mark
>
>
> On Thu, Feb 7, 2013 at 4:15 PM, Shmuel Krakower <sh...@gmail.com>
> wrote:
>
> >
> >
> http://jmeter.512774.n5.nabble.com/exiting-with-an-error-return-code-when-Assertion-Test-fails-td5715096.html
> >
> > ("....you can use the "System.exit(ERROR_CODE);" inside a beanshell
> > sampler.")
> >
> > Shmuel Krakower.
> > www.Beatsoo.org - re-use your jmeter scripts for application performance
> > monitoring from worldwide locations for free.
> >
> >
> > On Thu, Feb 7, 2013 at 9:09 PM, Mark Miller <ma...@gmail.com>
> > wrote:
> >
> > > I am in the process of integrating *discreet, functional unit
> > > tests*(leveraging several JMeter capabilities) with an enterprise
> > > monitoring
> > > system (Orion). The monitoring system can integrate with any executable
> > > based on exit status.
> > >
> > > I've done some searching and read other threads from folks seemingly
> > > searching for a similar capability.
> > >
> > > I've come up with and successfully tested the following solution and
> > wanted
> > > to float it to the community to ask for feedback or other suggestions:
> > >
> > > In the tesplan, I include a Generate Summary Results listener.
> > > I wrap the execution of JMeter in a simple shell script used to
> evaluate
> > > the string put to stdout by the listener:
> > >
> > > #!/bin/bash
> > > if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx |
> grep
> > "0
> > > (0.00%)$"
> > > then
> > > echo Success
> > > exit 0
> > > else
> > > echo Failure
> > > exit 1
> > > fi
> > >
> > > I have noted that, in a long running test, Generate Summary Results
> will
> > > output n+ lines over time. But it seems safe that the final line output
> > > terminates ($) with the final overall error rate.
> > >
> > > Anyone have any other cautions / considerations / approaches to this?
> > >
> > > Thank you.
> > >
> > > Mark
> > >
> >
>
Re: Exit code as indicator of (functional) test outcome
Posted by Mark Miller <ma...@gmail.com>.
Thanks Shmuel. I did find that suggestion that you'd made on that thread
and I experimented with it. It did work. However, I wondered that it might
be overly 'abrupt' in terms of JMeter exiting and finishing writing other
listeners (file handling), etc. I have not taken the time to look into
this, though. The strength of that approach is that it is more portable.
Do you happen to know how JMeter handles it's own termination under a
System.exit via BeanShell?
Mark
On Thu, Feb 7, 2013 at 4:15 PM, Shmuel Krakower <sh...@gmail.com> wrote:
>
> http://jmeter.512774.n5.nabble.com/exiting-with-an-error-return-code-when-Assertion-Test-fails-td5715096.html
>
> ("....you can use the "System.exit(ERROR_CODE);" inside a beanshell
> sampler.")
>
> Shmuel Krakower.
> www.Beatsoo.org - re-use your jmeter scripts for application performance
> monitoring from worldwide locations for free.
>
>
> On Thu, Feb 7, 2013 at 9:09 PM, Mark Miller <ma...@gmail.com>
> wrote:
>
> > I am in the process of integrating *discreet, functional unit
> > tests*(leveraging several JMeter capabilities) with an enterprise
> > monitoring
> > system (Orion). The monitoring system can integrate with any executable
> > based on exit status.
> >
> > I've done some searching and read other threads from folks seemingly
> > searching for a similar capability.
> >
> > I've come up with and successfully tested the following solution and
> wanted
> > to float it to the community to ask for feedback or other suggestions:
> >
> > In the tesplan, I include a Generate Summary Results listener.
> > I wrap the execution of JMeter in a simple shell script used to evaluate
> > the string put to stdout by the listener:
> >
> > #!/bin/bash
> > if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx | grep
> "0
> > (0.00%)$"
> > then
> > echo Success
> > exit 0
> > else
> > echo Failure
> > exit 1
> > fi
> >
> > I have noted that, in a long running test, Generate Summary Results will
> > output n+ lines over time. But it seems safe that the final line output
> > terminates ($) with the final overall error rate.
> >
> > Anyone have any other cautions / considerations / approaches to this?
> >
> > Thank you.
> >
> > Mark
> >
>
Re: Exit code as indicator of (functional) test outcome
Posted by Shmuel Krakower <sh...@gmail.com>.
http://jmeter.512774.n5.nabble.com/exiting-with-an-error-return-code-when-Assertion-Test-fails-td5715096.html
("....you can use the "System.exit(ERROR_CODE);" inside a beanshell
sampler.")
Shmuel Krakower.
www.Beatsoo.org - re-use your jmeter scripts for application performance
monitoring from worldwide locations for free.
On Thu, Feb 7, 2013 at 9:09 PM, Mark Miller <ma...@gmail.com> wrote:
> I am in the process of integrating *discreet, functional unit
> tests*(leveraging several JMeter capabilities) with an enterprise
> monitoring
> system (Orion). The monitoring system can integrate with any executable
> based on exit status.
>
> I've done some searching and read other threads from folks seemingly
> searching for a similar capability.
>
> I've come up with and successfully tested the following solution and wanted
> to float it to the community to ask for feedback or other suggestions:
>
> In the tesplan, I include a Generate Summary Results listener.
> I wrap the execution of JMeter in a simple shell script used to evaluate
> the string put to stdout by the listener:
>
> #!/bin/bash
> if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx | grep "0
> (0.00%)$"
> then
> echo Success
> exit 0
> else
> echo Failure
> exit 1
> fi
>
> I have noted that, in a long running test, Generate Summary Results will
> output n+ lines over time. But it seems safe that the final line output
> terminates ($) with the final overall error rate.
>
> Anyone have any other cautions / considerations / approaches to this?
>
> Thank you.
>
> Mark
>
Re: Exit code as indicator of (functional) test outcome
Posted by Mark Miller <ma...@gmail.com>.
Thanks Sergio, Sebb, Shmuel,
Sergio - it sounds like you settled on a similar approach to the one I have
used with the exception that you are taking responsibility for parsing the
test results to determine your outcome. I'm letting Generate Summary Report
handle that aggregate outcome and grepping for the final percentage - so
that seems like it all shakes out the same.
Sebb - thanks for weighing in. I figured that there was not much that the
program could do about the JVM exiting.
Shmuel - for my own out of curiosity, I ran a test where one listener was
writing XML to disk while a System.exit(1) happened randomly within the
test. As expected, the filewriting was dropped midstream (so the XML was
malformed) but, otherwise, I noticed no other issues or negative effects.
Thanks again for your thoughts on this.
Cheers,
Mark
On Fri, Feb 8, 2013 at 9:08 AM, Sergio Boso <se...@bosoconsulting.it>wrote:
> HI Mark,
>
> based on previous work on a Nagios plugin, I develop a Perl script to do
> that.
>
> The script behave like this:
> - launch jmeter with a proper script
> - analyse .csv result, looking for sampler errors, logical errors
> (assertions) and time thresholds on specific transactions.
>
> Then, it returns performance data in a Nagios-compliant way.
> It is not published yet (although I should do that, sooner or later), but
> I can send it to you...
>
> regards
>
> Sergio
> Il 07/02/2013 20:09, Mark Miller ha scritto:
>
>> I am in the process of integrating *discreet, functional unit
>> tests*(leveraging several JMeter capabilities) with an enterprise
>>
>> monitoring
>> system (Orion). The monitoring system can integrate with any executable
>> based on exit status.
>>
>> I've done some searching and read other threads from folks seemingly
>> searching for a similar capability.
>>
>> I've come up with and successfully tested the following solution and
>> wanted
>> to float it to the community to ask for feedback or other suggestions:
>>
>> In the tesplan, I include a Generate Summary Results listener.
>> I wrap the execution of JMeter in a simple shell script used to evaluate
>> the string put to stdout by the listener:
>>
>> #!/bin/bash
>> if java -jar ./apache-jmeter-2.9/bin/**ApacheJMeter.jar -n -t x.jmx |
>> grep "0
>> (0.00%)$"
>> then
>> echo Success
>> exit 0
>> else
>> echo Failure
>> exit 1
>> fi
>>
>> I have noted that, in a long running test, Generate Summary Results will
>> output n+ lines over time. But it seems safe that the final line output
>> terminates ($) with the final overall error rate.
>>
>> Anyone have any other cautions / considerations / approaches to this?
>>
>> Thank you.
>>
>> Mark
>>
>>
>
> --
>
> Ing. Sergio Boso
>
>
>
>
--
Mark Miller
Re: Exit code as indicator of (functional) test outcome
Posted by Sergio Boso <se...@bosoconsulting.it>.
HI Mark,
based on previous work on a Nagios plugin, I develop a Perl script to do that.
The script behave like this:
- launch jmeter with a proper script
- analyse .csv result, looking for sampler errors, logical errors (assertions) and time thresholds on specific transactions.
Then, it returns performance data in a Nagios-compliant way.
It is not published yet (although I should do that, sooner or later), but I can send it to you...
regards
Sergio
Il 07/02/2013 20:09, Mark Miller ha scritto:
> I am in the process of integrating *discreet, functional unit
> tests*(leveraging several JMeter capabilities) with an enterprise
> monitoring
> system (Orion). The monitoring system can integrate with any executable
> based on exit status.
>
> I've done some searching and read other threads from folks seemingly
> searching for a similar capability.
>
> I've come up with and successfully tested the following solution and wanted
> to float it to the community to ask for feedback or other suggestions:
>
> In the tesplan, I include a Generate Summary Results listener.
> I wrap the execution of JMeter in a simple shell script used to evaluate
> the string put to stdout by the listener:
>
> #!/bin/bash
> if java -jar ./apache-jmeter-2.9/bin/ApacheJMeter.jar -n -t x.jmx | grep "0
> (0.00%)$"
> then
> echo Success
> exit 0
> else
> echo Failure
> exit 1
> fi
>
> I have noted that, in a long running test, Generate Summary Results will
> output n+ lines over time. But it seems safe that the final line output
> terminates ($) with the final overall error rate.
>
> Anyone have any other cautions / considerations / approaches to this?
>
> Thank you.
>
> Mark
>
--
Ing. Sergio Boso
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org