You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Alexei Fedotov <al...@gmail.com> on 2007/06/21 16:53:55 UTC
[buildtest][harness] performance measurement infrastructure
Hello Vladimir,
Could yout please give few hints how one should write performance
tests using the harness? I found an option "-performance" but cannot
see anything else. I wonder how a testcase could report a performance
metrics.
Actually I'm thinking now of reviving comparative stress testing approach.
[1] http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
> Thanks for your explanations.
>
> Let me address the main problem I see to migrate a stress test suite
> to p-unit. To which extend may I count on your support at further
> p-unit development?
4 hours per day. :)
I listed very specific functionality which
> probably didn't suite well your original idea.
>
>
>
> On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > > This is very interesting. Thank you for the reference.
> >
> >
> > Hi Alexei, please see my comments inline
> >
> > I've got few questions trying to understand your framework.
> > > 1. Why an interface Watcher extends Serializable?
> >
> >
> > Because it's for performance comparison between different vms. The
> runner
> > will be serialized, so that the vm can load the right runner. You can
> > consider it like serializing the configuration for the vm.
> >
> > 2. Is there any project which uses p-unit for testing? Why have you
> started
> > > it?
> >
> >
> > Polepos (https://sourceforge.net/projects/polepos) is going to refactor
> its
> > code to use p-unit.
> >
> > And I know serveral community members use p-unit to test concurrency
> > performance(by running the existing JUnit tests).
> >
> > I started it because I found that I can't do something as I want with
> JUnit,
> > :) like concurrency, performance test. Why do I have to start thread,
> and
> > join, and check the result at the end of join everytime? why do I need
> to
> > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> >
> > 3. Running tests in several VM is great, but how could I compare these
> > > results?
> >
> >
> > There're serveral build-in reports in p-unit. The most straightforward
> one
> > is pdf format, which contains performance chart. The results of
> different vm
> > for the same method are shown in the same page, so that you can know
> which
> > one is better easily. It's also easy to plugin your own report by
> > implementing PUnitEventListener.
> >
> > 4. If I need to increase a parameter until the test fails and then
> > > report the final value as a test performance metrics how this should
> > > be done in p-unit?
> >
> >
> > hm... this functionality is not supported in p-unit....
> >
> > Thanks!
> >
> > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Hello Vladimir,
> > > > >
> > > > > Could yout please give few hints how one should write performance
> > > > > tests using the harness? I found an option "-performance" but
> cannot
> > > > > see anything else. I wonder how a testcase could report a
> performance
> > > > > metrics.
> > > > >
> > > > > Actually I'm thinking now of reviving comparative stress testing
> > > approach.
> > > >
> > > >
> > > > Hi Alexei,
> > > >
> > > > Do you have a chance to look at p-unit, (
> http://p-unit.sourceforge.net/
> > > )?
> > > >
> > > > I think it fulfills most of the requests listed in the documents.
> > > >
> > > > Another advantage of p-unit is that it can compare the performance
> > > between
> > > > different vms.
> > > >
> > > > One problem may be the license (GPL now, going to change the license
> to
> > > GPL
> > > > v3). If we really take it serious, I can relicense it immediately.
> > > >
> > > > [1]
> > > > >
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
>
> I haven't tried demo yet. I've noticed that you added a test for new
> functionality and decided to try it. I looked for
> trunk/punit.test/README.txt, then tried to find test targets in
> build.xml files. Finally I've tried Eclipse test runner:
>
> 1. Started Eclipse with the workspace set to trunk/
> 2. Added punit, punit.extension, punit.samples, punit.test to the project
> tree.
> 3. Built punit and punit.extension by means of Ant.
> 4. Pressed right mouse button over punit.test and selected Run as -> JUnit
> test
Hi Alexei,
You can select tests.api.org.punit.all.AllTests, and select Run As -> JUnit
test to run all tests by JUnit, or select
tests.api.org.punit.all.PUnitAllTests and select Run As -> Java Application
to run all tests by p-unit.
5. Among 113 testcases 6 testcases testcases of
> tests.api.org.punit.testclasses.JUnitAnnotationTestClass reported an
> error.
It's a "bug"... the problem of method signature. :)
6. Also I see that tests.api.org.punit.testclasses.LoopTestClass was
> absent in the test run results.
SoloRunnerTest#testRunLoopTest and ConcurrentRunnerTest#testRunLoopTest test
the LoopTestClass. LoopTestSample in punit.samples shows how to run a Loop
test.
When I ran both problematic classes as standalone applications I could
> see that testcases pass. That's make me think that the test design is
> ok, but it is not aligned with what Eclipse expects.
>
> To summarize, I see errors and missed tests. This might be a bug, or
> just I don't know enough details on how to run all tests.
It's because of my poor documentation. Sorry! Thanks!
On 6/26/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > >
> > > That's great. You are really quick. I was still thinking when your
> > > solution came up. I was able to build the project and run tests one by
> > > one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> > > tests. Probably it could make sense to use just an ordinary junit for
> > > testing p-unit itself.
> >
> >
> > Hi Alexei,
> >
> > Do you mean p-unit tests here? p-unit uses both ordinary junit 4 and
> p-unit
> > itself to run all tests for p-unit. They're AllTests(JUnit test suite
> > format) and PUnitAllTests(p-unit test suite format) under
> > punit.test\src\tests\api\org\punit\all directory.
> > Files in punit.sample are run one by one, because each file is supposed
> to
> > demo one functionality. :) It's for demo (not testing).
> >
> > Thanks!
> >
> > <snip>
> > > On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Andrew,
> > > > > Thanks for your explanations.
> > > > >
> > > > > Let me address the main problem I see to migrate a stress test
> suite
> > > > > to p-unit. To which extend may I count on your support at further
> > > > > p-unit development? I listed very specific functionality which
> > > > > probably didn't suite well your original idea.
> > > >
> > > >
> > > > Hi Alexei,
> > > >
> > > > Would you please check the latest sample code samples.LoopTestSamplein
> > > > punit.sample project? Is it the same as the original idea of toStop,
> > > > toAbort? Thanks!
> > > >
> > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > >
> > > > > > > Andrew,
> > > > > > > This is very interesting. Thank you for the reference.
> > > > > >
> > > > > >
> > > > > > Hi Alexei, please see my comments inline
> > > > > >
> > > > > > I've got few questions trying to understand your framework.
> > > > > > > 1. Why an interface Watcher extends Serializable?
> > > > > >
> > > > > >
> > > > > > Because it's for performance comparison between different vms.
> The
> > > > > runner
> > > > > > will be serialized, so that the vm can load the right runner.
> You
> > > can
> > > > > > consider it like serializing the configuration for the vm.
> > > > > >
> > > > > > 2. Is there any project which uses p-unit for testing? Why have
> you
> > > > > started
> > > > > > > it?
> > > > > >
> > > > > >
> > > > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> > > refactor
> > > > > its
> > > > > > code to use p-unit.
> > > > > >
> > > > > > And I know serveral community members use p-unit to test
> concurrency
> > > > > > performance(by running the existing JUnit tests).
> > > > > >
> > > > > > I started it because I found that I can't do something as I want
> > > with
> > > > > JUnit,
> > > > > > :) like concurrency, performance test. Why do I have to start
> > > thread,
> > > > > and
> > > > > > join, and check the result at the end of join everytime? why do
> I
> > > need
> > > > > to
> > > > > > use System.currentMilliseconds and sysout... :) So I wrote
> p-unit.
> > > > > >
> > > > > > 3. Running tests in several VM is great, but how could I compare
> > > these
> > > > > > > results?
> > > > > >
> > > > > >
> > > > > > There're serveral build-in reports in p-unit. The most
> > > straightforward
> > > > > one
> > > > > > is pdf format, which contains performance chart. The results of
> > > > > different vm
> > > > > > for the same method are shown in the same page, so that you can
> know
> > > > > which
> > > > > > one is better easily. It's also easy to plugin your own report
> by
> > > > > > implementing PUnitEventListener.
> > > > > >
> > > > > > 4. If I need to increase a parameter until the test fails and
> then
> > > > > > > report the final value as a test performance metrics how this
> > > should
> > > > > > > be done in p-unit?
> > > > > >
> > > > > >
> > > > > > hm... this functionality is not supported in p-unit....
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hello Vladimir,
> > > > > > > > >
> > > > > > > > > Could yout please give few hints how one should write
> > > performance
> > > > > > > > > tests using the harness? I found an option "-performance"
> but
> > > > > cannot
> > > > > > > > > see anything else. I wonder how a testcase could report a
> > > > > performance
> > > > > > > > > metrics.
> > > > > > > > >
> > > > > > > > > Actually I'm thinking now of reviving comparative stress
> > > testing
> > > > > > > approach.
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Alexei,
> > > > > > > >
> > > > > > > > Do you have a chance to look at p-unit, (
> > > > > http://p-unit.sourceforge.net/
> > > > > > > )?
> > > > > > > >
> > > > > > > > I think it fulfills most of the requests listed in the
> > > documents.
> > > > > > > >
> > > > > > > > Another advantage of p-unit is that it can compare the
> > > performance
> > > > > > > between
> > > > > > > > different vms.
> > > > > > > >
> > > > > > > > One problem may be the license (GPL now, going to change the
> > > license
> > > > > to
> > > > > > > GPL
> > > > > > > > v3). If we really take it serious, I can relicense it
> > > immediately.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > With best regards,
> > > > > > > > > Alexei,
> > > > > > > > > ESSD, Intel
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Andrew Zhang
> > > > > > > >
> > > > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > With best regards,
> > > > > > > Alexei,
> > > > > > > ESSD, Intel
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew Zhang
> > > > > >
> > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew,
I haven't tried demo yet. I've noticed that you added a test for new
functionality and decided to try it. I looked for
trunk/punit.test/README.txt, then tried to find test targets in
build.xml files. Finally I've tried Eclipse test runner:
1. Started Eclipse with the workspace set to trunk/
2. Added punit, punit.extension, punit.samples, punit.test to the project tree.
3. Built punit and punit.extension by means of Ant.
4. Pressed right mouse button over punit.test and selected Run as -> JUnit test
5. Among 113 testcases 6 testcases testcases of
tests.api.org.punit.testclasses.JUnitAnnotationTestClass reported an
error.
6. Also I see that tests.api.org.punit.testclasses.LoopTestClass was
absent in the test run results.
When I ran both problematic classes as standalone applications I could
see that testcases pass. That's make me think that the test design is
ok, but it is not aligned with what Eclipse expects.
To summarize, I see errors and missed tests. This might be a bug, or
just I don't know enough details on how to run all tests.
On 6/26/07, Andrew Zhang <zh...@gmail.com> wrote:
> On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
> >
> > Andrew,
> >
> > That's great. You are really quick. I was still thinking when your
> > solution came up. I was able to build the project and run tests one by
> > one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> > tests. Probably it could make sense to use just an ordinary junit for
> > testing p-unit itself.
>
>
> Hi Alexei,
>
> Do you mean p-unit tests here? p-unit uses both ordinary junit 4 and p-unit
> itself to run all tests for p-unit. They're AllTests(JUnit test suite
> format) and PUnitAllTests(p-unit test suite format) under
> punit.test\src\tests\api\org\punit\all directory.
> Files in punit.sample are run one by one, because each file is supposed to
> demo one functionality. :) It's for demo (not testing).
>
> Thanks!
>
> <snip>
> > On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > >
> > > > Andrew,
> > > > Thanks for your explanations.
> > > >
> > > > Let me address the main problem I see to migrate a stress test suite
> > > > to p-unit. To which extend may I count on your support at further
> > > > p-unit development? I listed very specific functionality which
> > > > probably didn't suite well your original idea.
> > >
> > >
> > > Hi Alexei,
> > >
> > > Would you please check the latest sample code samples.LoopTestSample in
> > > punit.sample project? Is it the same as the original idea of toStop,
> > > toAbort? Thanks!
> > >
> > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > >
> > > > > > Andrew,
> > > > > > This is very interesting. Thank you for the reference.
> > > > >
> > > > >
> > > > > Hi Alexei, please see my comments inline
> > > > >
> > > > > I've got few questions trying to understand your framework.
> > > > > > 1. Why an interface Watcher extends Serializable?
> > > > >
> > > > >
> > > > > Because it's for performance comparison between different vms. The
> > > > runner
> > > > > will be serialized, so that the vm can load the right runner. You
> > can
> > > > > consider it like serializing the configuration for the vm.
> > > > >
> > > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > > started
> > > > > > it?
> > > > >
> > > > >
> > > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> > refactor
> > > > its
> > > > > code to use p-unit.
> > > > >
> > > > > And I know serveral community members use p-unit to test concurrency
> > > > > performance(by running the existing JUnit tests).
> > > > >
> > > > > I started it because I found that I can't do something as I want
> > with
> > > > JUnit,
> > > > > :) like concurrency, performance test. Why do I have to start
> > thread,
> > > > and
> > > > > join, and check the result at the end of join everytime? why do I
> > need
> > > > to
> > > > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > > > >
> > > > > 3. Running tests in several VM is great, but how could I compare
> > these
> > > > > > results?
> > > > >
> > > > >
> > > > > There're serveral build-in reports in p-unit. The most
> > straightforward
> > > > one
> > > > > is pdf format, which contains performance chart. The results of
> > > > different vm
> > > > > for the same method are shown in the same page, so that you can know
> > > > which
> > > > > one is better easily. It's also easy to plugin your own report by
> > > > > implementing PUnitEventListener.
> > > > >
> > > > > 4. If I need to increase a parameter until the test fails and then
> > > > > > report the final value as a test performance metrics how this
> > should
> > > > > > be done in p-unit?
> > > > >
> > > > >
> > > > > hm... this functionality is not supported in p-unit....
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hello Vladimir,
> > > > > > > >
> > > > > > > > Could yout please give few hints how one should write
> > performance
> > > > > > > > tests using the harness? I found an option "-performance" but
> > > > cannot
> > > > > > > > see anything else. I wonder how a testcase could report a
> > > > performance
> > > > > > > > metrics.
> > > > > > > >
> > > > > > > > Actually I'm thinking now of reviving comparative stress
> > testing
> > > > > > approach.
> > > > > > >
> > > > > > >
> > > > > > > Hi Alexei,
> > > > > > >
> > > > > > > Do you have a chance to look at p-unit, (
> > > > http://p-unit.sourceforge.net/
> > > > > > )?
> > > > > > >
> > > > > > > I think it fulfills most of the requests listed in the
> > documents.
> > > > > > >
> > > > > > > Another advantage of p-unit is that it can compare the
> > performance
> > > > > > between
> > > > > > > different vms.
> > > > > > >
> > > > > > > One problem may be the license (GPL now, going to change the
> > license
> > > > to
> > > > > > GPL
> > > > > > > v3). If we really take it serious, I can relicense it
> > immediately.
> > > > > > >
> > > > > > > [1]
> > > > > > > >
> > > > > >
> > > >
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > > >
> > > > > > > > --
> > > > > > > > With best regards,
> > > > > > > > Alexei,
> > > > > > > > ESSD, Intel
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Andrew Zhang
> > > > > > >
> > > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards,
> > > > > > Alexei,
> > > > > > ESSD, Intel
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew Zhang
> > > > >
> > > > > http://zhanghuangzhu.blogspot.com/
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew Zhang
> > >
> > > http://zhanghuangzhu.blogspot.com/
> > >
> >
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
>
> That's great. You are really quick. I was still thinking when your
> solution came up. I was able to build the project and run tests one by
> one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> tests. Probably it could make sense to use just an ordinary junit for
> testing p-unit itself.
Hi Alexei,
Do you mean p-unit tests here? p-unit uses both ordinary junit 4 and p-unit
itself to run all tests for p-unit. They're AllTests(JUnit test suite
format) and PUnitAllTests(p-unit test suite format) under
punit.test\src\tests\api\org\punit\all directory.
Files in punit.sample are run one by one, because each file is supposed to
demo one functionality. :) It's for demo (not testing).
Thanks!
<snip>
> On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > > Thanks for your explanations.
> > >
> > > Let me address the main problem I see to migrate a stress test suite
> > > to p-unit. To which extend may I count on your support at further
> > > p-unit development? I listed very specific functionality which
> > > probably didn't suite well your original idea.
> >
> >
> > Hi Alexei,
> >
> > Would you please check the latest sample code samples.LoopTestSample in
> > punit.sample project? Is it the same as the original idea of toStop,
> > toAbort? Thanks!
> >
> > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Andrew,
> > > > > This is very interesting. Thank you for the reference.
> > > >
> > > >
> > > > Hi Alexei, please see my comments inline
> > > >
> > > > I've got few questions trying to understand your framework.
> > > > > 1. Why an interface Watcher extends Serializable?
> > > >
> > > >
> > > > Because it's for performance comparison between different vms. The
> > > runner
> > > > will be serialized, so that the vm can load the right runner. You
> can
> > > > consider it like serializing the configuration for the vm.
> > > >
> > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > started
> > > > > it?
> > > >
> > > >
> > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> refactor
> > > its
> > > > code to use p-unit.
> > > >
> > > > And I know serveral community members use p-unit to test concurrency
> > > > performance(by running the existing JUnit tests).
> > > >
> > > > I started it because I found that I can't do something as I want
> with
> > > JUnit,
> > > > :) like concurrency, performance test. Why do I have to start
> thread,
> > > and
> > > > join, and check the result at the end of join everytime? why do I
> need
> > > to
> > > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > > >
> > > > 3. Running tests in several VM is great, but how could I compare
> these
> > > > > results?
> > > >
> > > >
> > > > There're serveral build-in reports in p-unit. The most
> straightforward
> > > one
> > > > is pdf format, which contains performance chart. The results of
> > > different vm
> > > > for the same method are shown in the same page, so that you can know
> > > which
> > > > one is better easily. It's also easy to plugin your own report by
> > > > implementing PUnitEventListener.
> > > >
> > > > 4. If I need to increase a parameter until the test fails and then
> > > > > report the final value as a test performance metrics how this
> should
> > > > > be done in p-unit?
> > > >
> > > >
> > > > hm... this functionality is not supported in p-unit....
> > > >
> > > > Thanks!
> > > >
> > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hello Vladimir,
> > > > > > >
> > > > > > > Could yout please give few hints how one should write
> performance
> > > > > > > tests using the harness? I found an option "-performance" but
> > > cannot
> > > > > > > see anything else. I wonder how a testcase could report a
> > > performance
> > > > > > > metrics.
> > > > > > >
> > > > > > > Actually I'm thinking now of reviving comparative stress
> testing
> > > > > approach.
> > > > > >
> > > > > >
> > > > > > Hi Alexei,
> > > > > >
> > > > > > Do you have a chance to look at p-unit, (
> > > http://p-unit.sourceforge.net/
> > > > > )?
> > > > > >
> > > > > > I think it fulfills most of the requests listed in the
> documents.
> > > > > >
> > > > > > Another advantage of p-unit is that it can compare the
> performance
> > > > > between
> > > > > > different vms.
> > > > > >
> > > > > > One problem may be the license (GPL now, going to change the
> license
> > > to
> > > > > GPL
> > > > > > v3). If we really take it serious, I can relicense it
> immediately.
> > > > > >
> > > > > > [1]
> > > > > > >
> > > > >
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > >
> > > > > > > --
> > > > > > > With best regards,
> > > > > > > Alexei,
> > > > > > > ESSD, Intel
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew Zhang
> > > > > >
> > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew, let me quickly add one more point. As far as I understand
toAbort() support is not implemented in punit for now. That's ok for
now. Let me make some proof of concept first.
On 6/28/07, Alexei Fedotov <al...@gmail.com> wrote:
> Andrew,
>
> You wrote:
> ------
> I must miss something here. For which scenario(Loop or Concurrent?), do we
> need toAbort? As my understanding, tearDown will stop the test case
> gracefully. Do we need this special setting? (But I saw the same requirement
> before, because the user's logic code is dependent of timestamp, and the
> test runner have to be slow enough to run it concurrently :) )
> ------
>
> I'm currently looking into your implementation of timers and now is
> able understand your question about my question. :-)
>
> If I understand correctly tearDown() is a method which is called by
> the framework when the test finishes execution of all test cases. In
> the case when we need toAbort setting this is not the case.
>
> The typical behavior for DRLVM is to deadlock on some stress test. If
> test deadlocks, it's ok, it means that test successfully found a bug
> in VM which should be fixed. So toAbort option has no relation to a
> particular test.
>
> From the other side when we run a number of tests in a sequence we
> need to continue execution of other tests after the test deadlocks.
> This could be done using a small amount of knowledge of VM
> architecture which says that java thread map to OS threads, and if we
> start a java thread which doesn't synchronize with other threads it
> has a good chance to survive after deadlock of test threads happens.
> This thread can either call System.exit() or kill the whole thread
> group where the threads created by the test are executed.
>
> If punit starts tests in separate JVMs, these JVMs can be also
> terminated externally, and this gives us even the third timer.
>
> Does it make sense?
>
> Thanks, Alexei
>
> On 6/26/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > >
> > > That's great. You are really quick. I was still thinking when your
> > > solution came up. I was able to build the project and run tests one by
> > > one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> > > tests. Probably it could make sense to use just an ordinary junit for
> > > testing p-unit itself.
> > >
> > > 1. I've just realized that I cannot link stress tests with p-unit
> > > while it is GPLed. So it's better to change a license. GPL with
> > > exception is ok, but it's safer to have BSD-like license. Actually
> > > this is the only comment which prevents me from using punit, all
> > > others are mostly for better understanding of your concept.
> >
> >
> > Hi Alexei,
> >
> > No problem. I'll change the license for the next release!
> >
> > AL2.0 should be safe, right?
> >
> > 2. Do you think "toStop" and "toAbort" from my requirements are the
> > > best terms? We can change them until there is not too much code
> > > developed. A typical use case is the following. I give a test a fair
> > > time to loop (= first timer) and a small type to stop gracefully (=
> > > second timer). The variant named them "toWork", "toAbort". Another
> > > variant which comes to my mind is "toLoop", "toStop". What do you
> > > think?
> >
> > 3. What do you think of using junit.extensions.TestDecorator for
> > > building complex tests on the top of the primitive ones?
> >
> >
> >
> > I must miss something here. For which scenario(Loop or Concurrent?), do we
> > need toAbort? As my understanding, tearDown will stop the test case
> > gracefully. Do we need this special setting? (But I saw the same requirement
> > before, because the user's logic code is dependent of timestamp, and the
> > test runner have to be slow enough to run it concurrently :) )
> >
> > Thanks!
> >
> >
> > On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Andrew,
> > > > > Thanks for your explanations.
> > > > >
> > > > > Let me address the main problem I see to migrate a stress test suite
> > > > > to p-unit. To which extend may I count on your support at further
> > > > > p-unit development? I listed very specific functionality which
> > > > > probably didn't suite well your original idea.
> > > >
> > > >
> > > > Hi Alexei,
> > > >
> > > > Would you please check the latest sample code samples.LoopTestSample in
> > > > punit.sample project? Is it the same as the original idea of toStop,
> > > > toAbort? Thanks!
> > > >
> > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > >
> > > > > > > Andrew,
> > > > > > > This is very interesting. Thank you for the reference.
> > > > > >
> > > > > >
> > > > > > Hi Alexei, please see my comments inline
> > > > > >
> > > > > > I've got few questions trying to understand your framework.
> > > > > > > 1. Why an interface Watcher extends Serializable?
> > > > > >
> > > > > >
> > > > > > Because it's for performance comparison between different vms. The
> > > > > runner
> > > > > > will be serialized, so that the vm can load the right runner. You
> > > can
> > > > > > consider it like serializing the configuration for the vm.
> > > > > >
> > > > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > > > started
> > > > > > > it?
> > > > > >
> > > > > >
> > > > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> > > refactor
> > > > > its
> > > > > > code to use p-unit.
> > > > > >
> > > > > > And I know serveral community members use p-unit to test concurrency
> > > > > > performance(by running the existing JUnit tests).
> > > > > >
> > > > > > I started it because I found that I can't do something as I want
> > > with
> > > > > JUnit,
> > > > > > :) like concurrency, performance test. Why do I have to start
> > > thread,
> > > > > and
> > > > > > join, and check the result at the end of join everytime? why do I
> > > need
> > > > > to
> > > > > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > > > > >
> > > > > > 3. Running tests in several VM is great, but how could I compare
> > > these
> > > > > > > results?
> > > > > >
> > > > > >
> > > > > > There're serveral build-in reports in p-unit. The most
> > > straightforward
> > > > > one
> > > > > > is pdf format, which contains performance chart. The results of
> > > > > different vm
> > > > > > for the same method are shown in the same page, so that you can know
> > > > > which
> > > > > > one is better easily. It's also easy to plugin your own report by
> > > > > > implementing PUnitEventListener.
> > > > > >
> > > > > > 4. If I need to increase a parameter until the test fails and then
> > > > > > > report the final value as a test performance metrics how this
> > > should
> > > > > > > be done in p-unit?
> > > > > >
> > > > > >
> > > > > > hm... this functionality is not supported in p-unit....
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hello Vladimir,
> > > > > > > > >
> > > > > > > > > Could yout please give few hints how one should write
> > > performance
> > > > > > > > > tests using the harness? I found an option "-performance" but
> > > > > cannot
> > > > > > > > > see anything else. I wonder how a testcase could report a
> > > > > performance
> > > > > > > > > metrics.
> > > > > > > > >
> > > > > > > > > Actually I'm thinking now of reviving comparative stress
> > > testing
> > > > > > > approach.
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Alexei,
> > > > > > > >
> > > > > > > > Do you have a chance to look at p-unit, (
> > > > > http://p-unit.sourceforge.net/
> > > > > > > )?
> > > > > > > >
> > > > > > > > I think it fulfills most of the requests listed in the
> > > documents.
> > > > > > > >
> > > > > > > > Another advantage of p-unit is that it can compare the
> > > performance
> > > > > > > between
> > > > > > > > different vms.
> > > > > > > >
> > > > > > > > One problem may be the license (GPL now, going to change the
> > > license
> > > > > to
> > > > > > > GPL
> > > > > > > > v3). If we really take it serious, I can relicense it
> > > immediately.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > > >
> > > > > > >
> > > > >
> > > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > With best regards,
> > > > > > > > > Alexei,
> > > > > > > > > ESSD, Intel
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Andrew Zhang
> > > > > > > >
> > > > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > With best regards,
> > > > > > > Alexei,
> > > > > > > ESSD, Intel
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew Zhang
> > > > > >
> > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew,
You wrote:
------
I must miss something here. For which scenario(Loop or Concurrent?), do we
need toAbort? As my understanding, tearDown will stop the test case
gracefully. Do we need this special setting? (But I saw the same requirement
before, because the user's logic code is dependent of timestamp, and the
test runner have to be slow enough to run it concurrently :) )
------
I'm currently looking into your implementation of timers and now is
able understand your question about my question. :-)
If I understand correctly tearDown() is a method which is called by
the framework when the test finishes execution of all test cases. In
the case when we need toAbort setting this is not the case.
The typical behavior for DRLVM is to deadlock on some stress test. If
test deadlocks, it's ok, it means that test successfully found a bug
in VM which should be fixed. So toAbort option has no relation to a
particular test.
>From the other side when we run a number of tests in a sequence we
need to continue execution of other tests after the test deadlocks.
This could be done using a small amount of knowledge of VM
architecture which says that java thread map to OS threads, and if we
start a java thread which doesn't synchronize with other threads it
has a good chance to survive after deadlock of test threads happens.
This thread can either call System.exit() or kill the whole thread
group where the threads created by the test are executed.
If punit starts tests in separate JVMs, these JVMs can be also
terminated externally, and this gives us even the third timer.
Does it make sense?
Thanks, Alexei
On 6/26/07, Andrew Zhang <zh...@gmail.com> wrote:
> On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
> >
> > Andrew,
> >
> > That's great. You are really quick. I was still thinking when your
> > solution came up. I was able to build the project and run tests one by
> > one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> > tests. Probably it could make sense to use just an ordinary junit for
> > testing p-unit itself.
> >
> > 1. I've just realized that I cannot link stress tests with p-unit
> > while it is GPLed. So it's better to change a license. GPL with
> > exception is ok, but it's safer to have BSD-like license. Actually
> > this is the only comment which prevents me from using punit, all
> > others are mostly for better understanding of your concept.
>
>
> Hi Alexei,
>
> No problem. I'll change the license for the next release!
>
> AL2.0 should be safe, right?
>
> 2. Do you think "toStop" and "toAbort" from my requirements are the
> > best terms? We can change them until there is not too much code
> > developed. A typical use case is the following. I give a test a fair
> > time to loop (= first timer) and a small type to stop gracefully (=
> > second timer). The variant named them "toWork", "toAbort". Another
> > variant which comes to my mind is "toLoop", "toStop". What do you
> > think?
>
> 3. What do you think of using junit.extensions.TestDecorator for
> > building complex tests on the top of the primitive ones?
>
>
>
> I must miss something here. For which scenario(Loop or Concurrent?), do we
> need toAbort? As my understanding, tearDown will stop the test case
> gracefully. Do we need this special setting? (But I saw the same requirement
> before, because the user's logic code is dependent of timestamp, and the
> test runner have to be slow enough to run it concurrently :) )
>
> Thanks!
>
>
> On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > >
> > > > Andrew,
> > > > Thanks for your explanations.
> > > >
> > > > Let me address the main problem I see to migrate a stress test suite
> > > > to p-unit. To which extend may I count on your support at further
> > > > p-unit development? I listed very specific functionality which
> > > > probably didn't suite well your original idea.
> > >
> > >
> > > Hi Alexei,
> > >
> > > Would you please check the latest sample code samples.LoopTestSample in
> > > punit.sample project? Is it the same as the original idea of toStop,
> > > toAbort? Thanks!
> > >
> > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > >
> > > > > > Andrew,
> > > > > > This is very interesting. Thank you for the reference.
> > > > >
> > > > >
> > > > > Hi Alexei, please see my comments inline
> > > > >
> > > > > I've got few questions trying to understand your framework.
> > > > > > 1. Why an interface Watcher extends Serializable?
> > > > >
> > > > >
> > > > > Because it's for performance comparison between different vms. The
> > > > runner
> > > > > will be serialized, so that the vm can load the right runner. You
> > can
> > > > > consider it like serializing the configuration for the vm.
> > > > >
> > > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > > started
> > > > > > it?
> > > > >
> > > > >
> > > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> > refactor
> > > > its
> > > > > code to use p-unit.
> > > > >
> > > > > And I know serveral community members use p-unit to test concurrency
> > > > > performance(by running the existing JUnit tests).
> > > > >
> > > > > I started it because I found that I can't do something as I want
> > with
> > > > JUnit,
> > > > > :) like concurrency, performance test. Why do I have to start
> > thread,
> > > > and
> > > > > join, and check the result at the end of join everytime? why do I
> > need
> > > > to
> > > > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > > > >
> > > > > 3. Running tests in several VM is great, but how could I compare
> > these
> > > > > > results?
> > > > >
> > > > >
> > > > > There're serveral build-in reports in p-unit. The most
> > straightforward
> > > > one
> > > > > is pdf format, which contains performance chart. The results of
> > > > different vm
> > > > > for the same method are shown in the same page, so that you can know
> > > > which
> > > > > one is better easily. It's also easy to plugin your own report by
> > > > > implementing PUnitEventListener.
> > > > >
> > > > > 4. If I need to increase a parameter until the test fails and then
> > > > > > report the final value as a test performance metrics how this
> > should
> > > > > > be done in p-unit?
> > > > >
> > > > >
> > > > > hm... this functionality is not supported in p-unit....
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hello Vladimir,
> > > > > > > >
> > > > > > > > Could yout please give few hints how one should write
> > performance
> > > > > > > > tests using the harness? I found an option "-performance" but
> > > > cannot
> > > > > > > > see anything else. I wonder how a testcase could report a
> > > > performance
> > > > > > > > metrics.
> > > > > > > >
> > > > > > > > Actually I'm thinking now of reviving comparative stress
> > testing
> > > > > > approach.
> > > > > > >
> > > > > > >
> > > > > > > Hi Alexei,
> > > > > > >
> > > > > > > Do you have a chance to look at p-unit, (
> > > > http://p-unit.sourceforge.net/
> > > > > > )?
> > > > > > >
> > > > > > > I think it fulfills most of the requests listed in the
> > documents.
> > > > > > >
> > > > > > > Another advantage of p-unit is that it can compare the
> > performance
> > > > > > between
> > > > > > > different vms.
> > > > > > >
> > > > > > > One problem may be the license (GPL now, going to change the
> > license
> > > > to
> > > > > > GPL
> > > > > > > v3). If we really take it serious, I can relicense it
> > immediately.
> > > > > > >
> > > > > > > [1]
> > > > > > > >
> > > > > >
> > > >
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > > >
> > > > > > > > --
> > > > > > > > With best regards,
> > > > > > > > Alexei,
> > > > > > > > ESSD, Intel
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Andrew Zhang
> > > > > > >
> > > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards,
> > > > > > Alexei,
> > > > > > ESSD, Intel
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew Zhang
> > > > >
> > > > > http://zhanghuangzhu.blogspot.com/
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew Zhang
> > >
> > > http://zhanghuangzhu.blogspot.com/
> > >
> >
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness][p-unit] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/26/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
>
> That's great. You are really quick. I was still thinking when your
> solution came up. I was able to build the project and run tests one by
> one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> tests. Probably it could make sense to use just an ordinary junit for
> testing p-unit itself.
>
> 1. I've just realized that I cannot link stress tests with p-unit
> while it is GPLed. So it's better to change a license. GPL with
> exception is ok, but it's safer to have BSD-like license. Actually
> this is the only comment which prevents me from using punit, all
> others are mostly for better understanding of your concept.
Hi Alexei,
No problem. I'll change the license for the next release!
AL2.0 should be safe, right?
2. Do you think "toStop" and "toAbort" from my requirements are the
> best terms? We can change them until there is not too much code
> developed. A typical use case is the following. I give a test a fair
> time to loop (= first timer) and a small type to stop gracefully (=
> second timer). The variant named them "toWork", "toAbort". Another
> variant which comes to my mind is "toLoop", "toStop". What do you
> think?
3. What do you think of using junit.extensions.TestDecorator for
> building complex tests on the top of the primitive ones?
I must miss something here. For which scenario(Loop or Concurrent?), do we
need toAbort? As my understanding, tearDown will stop the test case
gracefully. Do we need this special setting? (But I saw the same requirement
before, because the user's logic code is dependent of timestamp, and the
test runner have to be slow enough to run it concurrently :) )
Thanks!
On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > > Thanks for your explanations.
> > >
> > > Let me address the main problem I see to migrate a stress test suite
> > > to p-unit. To which extend may I count on your support at further
> > > p-unit development? I listed very specific functionality which
> > > probably didn't suite well your original idea.
> >
> >
> > Hi Alexei,
> >
> > Would you please check the latest sample code samples.LoopTestSample in
> > punit.sample project? Is it the same as the original idea of toStop,
> > toAbort? Thanks!
> >
> > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Andrew,
> > > > > This is very interesting. Thank you for the reference.
> > > >
> > > >
> > > > Hi Alexei, please see my comments inline
> > > >
> > > > I've got few questions trying to understand your framework.
> > > > > 1. Why an interface Watcher extends Serializable?
> > > >
> > > >
> > > > Because it's for performance comparison between different vms. The
> > > runner
> > > > will be serialized, so that the vm can load the right runner. You
> can
> > > > consider it like serializing the configuration for the vm.
> > > >
> > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > started
> > > > > it?
> > > >
> > > >
> > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> refactor
> > > its
> > > > code to use p-unit.
> > > >
> > > > And I know serveral community members use p-unit to test concurrency
> > > > performance(by running the existing JUnit tests).
> > > >
> > > > I started it because I found that I can't do something as I want
> with
> > > JUnit,
> > > > :) like concurrency, performance test. Why do I have to start
> thread,
> > > and
> > > > join, and check the result at the end of join everytime? why do I
> need
> > > to
> > > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > > >
> > > > 3. Running tests in several VM is great, but how could I compare
> these
> > > > > results?
> > > >
> > > >
> > > > There're serveral build-in reports in p-unit. The most
> straightforward
> > > one
> > > > is pdf format, which contains performance chart. The results of
> > > different vm
> > > > for the same method are shown in the same page, so that you can know
> > > which
> > > > one is better easily. It's also easy to plugin your own report by
> > > > implementing PUnitEventListener.
> > > >
> > > > 4. If I need to increase a parameter until the test fails and then
> > > > > report the final value as a test performance metrics how this
> should
> > > > > be done in p-unit?
> > > >
> > > >
> > > > hm... this functionality is not supported in p-unit....
> > > >
> > > > Thanks!
> > > >
> > > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hello Vladimir,
> > > > > > >
> > > > > > > Could yout please give few hints how one should write
> performance
> > > > > > > tests using the harness? I found an option "-performance" but
> > > cannot
> > > > > > > see anything else. I wonder how a testcase could report a
> > > performance
> > > > > > > metrics.
> > > > > > >
> > > > > > > Actually I'm thinking now of reviving comparative stress
> testing
> > > > > approach.
> > > > > >
> > > > > >
> > > > > > Hi Alexei,
> > > > > >
> > > > > > Do you have a chance to look at p-unit, (
> > > http://p-unit.sourceforge.net/
> > > > > )?
> > > > > >
> > > > > > I think it fulfills most of the requests listed in the
> documents.
> > > > > >
> > > > > > Another advantage of p-unit is that it can compare the
> performance
> > > > > between
> > > > > > different vms.
> > > > > >
> > > > > > One problem may be the license (GPL now, going to change the
> license
> > > to
> > > > > GPL
> > > > > > v3). If we really take it serious, I can relicense it
> immediately.
> > > > > >
> > > > > > [1]
> > > > > > >
> > > > >
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > >
> > > > > > > --
> > > > > > > With best regards,
> > > > > > > Alexei,
> > > > > > > ESSD, Intel
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew Zhang
> > > > > >
> > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
[buildtest][harness][p-unit] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew,
That's great. You are really quick. I was still thinking when your
solution came up. I was able to build the project and run tests one by
one, though I wasn't able to use the perfect Eclipse plugin for JUnit
tests. Probably it could make sense to use just an ordinary junit for
testing p-unit itself.
1. I've just realized that I cannot link stress tests with p-unit
while it is GPLed. So it's better to change a license. GPL with
exception is ok, but it's safer to have BSD-like license. Actually
this is the only comment which prevents me from using punit, all
others are mostly for better understanding of your concept.
2. Do you think "toStop" and "toAbort" from my requirements are the
best terms? We can change them until there is not too much code
developed. A typical use case is the following. I give a test a fair
time to loop (= first timer) and a small type to stop gracefully (=
second timer). The variant named them "toWork", "toAbort". Another
variant which comes to my mind is "toLoop", "toStop". What do you
think?
3. What do you think of using junit.extensions.TestDecorator for
building complex tests on the top of the primitive ones?
On 6/25/07, Andrew Zhang <zh...@gmail.com> wrote:
> On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> >
> > Andrew,
> > Thanks for your explanations.
> >
> > Let me address the main problem I see to migrate a stress test suite
> > to p-unit. To which extend may I count on your support at further
> > p-unit development? I listed very specific functionality which
> > probably didn't suite well your original idea.
>
>
> Hi Alexei,
>
> Would you please check the latest sample code samples.LoopTestSample in
> punit.sample project? Is it the same as the original idea of toStop,
> toAbort? Thanks!
>
> On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > >
> > > > Andrew,
> > > > This is very interesting. Thank you for the reference.
> > >
> > >
> > > Hi Alexei, please see my comments inline
> > >
> > > I've got few questions trying to understand your framework.
> > > > 1. Why an interface Watcher extends Serializable?
> > >
> > >
> > > Because it's for performance comparison between different vms. The
> > runner
> > > will be serialized, so that the vm can load the right runner. You can
> > > consider it like serializing the configuration for the vm.
> > >
> > > 2. Is there any project which uses p-unit for testing? Why have you
> > started
> > > > it?
> > >
> > >
> > > Polepos (https://sourceforge.net/projects/polepos) is going to refactor
> > its
> > > code to use p-unit.
> > >
> > > And I know serveral community members use p-unit to test concurrency
> > > performance(by running the existing JUnit tests).
> > >
> > > I started it because I found that I can't do something as I want with
> > JUnit,
> > > :) like concurrency, performance test. Why do I have to start thread,
> > and
> > > join, and check the result at the end of join everytime? why do I need
> > to
> > > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> > >
> > > 3. Running tests in several VM is great, but how could I compare these
> > > > results?
> > >
> > >
> > > There're serveral build-in reports in p-unit. The most straightforward
> > one
> > > is pdf format, which contains performance chart. The results of
> > different vm
> > > for the same method are shown in the same page, so that you can know
> > which
> > > one is better easily. It's also easy to plugin your own report by
> > > implementing PUnitEventListener.
> > >
> > > 4. If I need to increase a parameter until the test fails and then
> > > > report the final value as a test performance metrics how this should
> > > > be done in p-unit?
> > >
> > >
> > > hm... this functionality is not supported in p-unit....
> > >
> > > Thanks!
> > >
> > > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > >
> > > > > > Hello Vladimir,
> > > > > >
> > > > > > Could yout please give few hints how one should write performance
> > > > > > tests using the harness? I found an option "-performance" but
> > cannot
> > > > > > see anything else. I wonder how a testcase could report a
> > performance
> > > > > > metrics.
> > > > > >
> > > > > > Actually I'm thinking now of reviving comparative stress testing
> > > > approach.
> > > > >
> > > > >
> > > > > Hi Alexei,
> > > > >
> > > > > Do you have a chance to look at p-unit, (
> > http://p-unit.sourceforge.net/
> > > > )?
> > > > >
> > > > > I think it fulfills most of the requests listed in the documents.
> > > > >
> > > > > Another advantage of p-unit is that it can compare the performance
> > > > between
> > > > > different vms.
> > > > >
> > > > > One problem may be the license (GPL now, going to change the license
> > to
> > > > GPL
> > > > > v3). If we really take it serious, I can relicense it immediately.
> > > > >
> > > > > [1]
> > > > > >
> > > >
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > >
> > > > > > --
> > > > > > With best regards,
> > > > > > Alexei,
> > > > > > ESSD, Intel
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew Zhang
> > > > >
> > > > > http://zhanghuangzhu.blogspot.com/
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew Zhang
> > >
> > > http://zhanghuangzhu.blogspot.com/
> > >
> >
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
> Thanks for your explanations.
>
> Let me address the main problem I see to migrate a stress test suite
> to p-unit. To which extend may I count on your support at further
> p-unit development? I listed very specific functionality which
> probably didn't suite well your original idea.
Hi Alexei,
Would you please check the latest sample code samples.LoopTestSample in
punit.sample project? Is it the same as the original idea of toStop,
toAbort? Thanks!
On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Andrew,
> > > This is very interesting. Thank you for the reference.
> >
> >
> > Hi Alexei, please see my comments inline
> >
> > I've got few questions trying to understand your framework.
> > > 1. Why an interface Watcher extends Serializable?
> >
> >
> > Because it's for performance comparison between different vms. The
> runner
> > will be serialized, so that the vm can load the right runner. You can
> > consider it like serializing the configuration for the vm.
> >
> > 2. Is there any project which uses p-unit for testing? Why have you
> started
> > > it?
> >
> >
> > Polepos (https://sourceforge.net/projects/polepos) is going to refactor
> its
> > code to use p-unit.
> >
> > And I know serveral community members use p-unit to test concurrency
> > performance(by running the existing JUnit tests).
> >
> > I started it because I found that I can't do something as I want with
> JUnit,
> > :) like concurrency, performance test. Why do I have to start thread,
> and
> > join, and check the result at the end of join everytime? why do I need
> to
> > use System.currentMilliseconds and sysout... :) So I wrote p-unit.
> >
> > 3. Running tests in several VM is great, but how could I compare these
> > > results?
> >
> >
> > There're serveral build-in reports in p-unit. The most straightforward
> one
> > is pdf format, which contains performance chart. The results of
> different vm
> > for the same method are shown in the same page, so that you can know
> which
> > one is better easily. It's also easy to plugin your own report by
> > implementing PUnitEventListener.
> >
> > 4. If I need to increase a parameter until the test fails and then
> > > report the final value as a test performance metrics how this should
> > > be done in p-unit?
> >
> >
> > hm... this functionality is not supported in p-unit....
> >
> > Thanks!
> >
> > On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > >
> > > > > Hello Vladimir,
> > > > >
> > > > > Could yout please give few hints how one should write performance
> > > > > tests using the harness? I found an option "-performance" but
> cannot
> > > > > see anything else. I wonder how a testcase could report a
> performance
> > > > > metrics.
> > > > >
> > > > > Actually I'm thinking now of reviving comparative stress testing
> > > approach.
> > > >
> > > >
> > > > Hi Alexei,
> > > >
> > > > Do you have a chance to look at p-unit, (
> http://p-unit.sourceforge.net/
> > > )?
> > > >
> > > > I think it fulfills most of the requests listed in the documents.
> > > >
> > > > Another advantage of p-unit is that it can compare the performance
> > > between
> > > > different vms.
> > > >
> > > > One problem may be the license (GPL now, going to change the license
> to
> > > GPL
> > > > v3). If we really take it serious, I can relicense it immediately.
> > > >
> > > > [1]
> > > > >
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew Zhang
> > > >
> > > > http://zhanghuangzhu.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
Re: [buildtest][harness] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew,
Thanks for your explanations.
Let me address the main problem I see to migrate a stress test suite
to p-unit. To which extend may I count on your support at further
p-unit development? I listed very specific functionality which
probably didn't suite well your original idea.
On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
> >
> > Andrew,
> > This is very interesting. Thank you for the reference.
>
>
> Hi Alexei, please see my comments inline
>
> I've got few questions trying to understand your framework.
> > 1. Why an interface Watcher extends Serializable?
>
>
> Because it's for performance comparison between different vms. The runner
> will be serialized, so that the vm can load the right runner. You can
> consider it like serializing the configuration for the vm.
>
> 2. Is there any project which uses p-unit for testing? Why have you started
> > it?
>
>
> Polepos (https://sourceforge.net/projects/polepos) is going to refactor its
> code to use p-unit.
>
> And I know serveral community members use p-unit to test concurrency
> performance(by running the existing JUnit tests).
>
> I started it because I found that I can't do something as I want with JUnit,
> :) like concurrency, performance test. Why do I have to start thread, and
> join, and check the result at the end of join everytime? why do I need to
> use System.currentMilliseconds and sysout... :) So I wrote p-unit.
>
> 3. Running tests in several VM is great, but how could I compare these
> > results?
>
>
> There're serveral build-in reports in p-unit. The most straightforward one
> is pdf format, which contains performance chart. The results of different vm
> for the same method are shown in the same page, so that you can know which
> one is better easily. It's also easy to plugin your own report by
> implementing PUnitEventListener.
>
> 4. If I need to increase a parameter until the test fails and then
> > report the final value as a test performance metrics how this should
> > be done in p-unit?
>
>
> hm... this functionality is not supported in p-unit....
>
> Thanks!
>
> On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > >
> > > > Hello Vladimir,
> > > >
> > > > Could yout please give few hints how one should write performance
> > > > tests using the harness? I found an option "-performance" but cannot
> > > > see anything else. I wonder how a testcase could report a performance
> > > > metrics.
> > > >
> > > > Actually I'm thinking now of reviving comparative stress testing
> > approach.
> > >
> > >
> > > Hi Alexei,
> > >
> > > Do you have a chance to look at p-unit, (http://p-unit.sourceforge.net/
> > )?
> > >
> > > I think it fulfills most of the requests listed in the documents.
> > >
> > > Another advantage of p-unit is that it can compare the performance
> > between
> > > different vms.
> > >
> > > One problem may be the license (GPL now, going to change the license to
> > GPL
> > > v3). If we really take it serious, I can relicense it immediately.
> > >
> > > [1]
> > > >
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew Zhang
> > >
> > > http://zhanghuangzhu.blogspot.com/
> > >
> >
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/22/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Andrew,
> This is very interesting. Thank you for the reference.
Hi Alexei, please see my comments inline
I've got few questions trying to understand your framework.
> 1. Why an interface Watcher extends Serializable?
Because it's for performance comparison between different vms. The runner
will be serialized, so that the vm can load the right runner. You can
consider it like serializing the configuration for the vm.
2. Is there any project which uses p-unit for testing? Why have you started
> it?
Polepos (https://sourceforge.net/projects/polepos) is going to refactor its
code to use p-unit.
And I know serveral community members use p-unit to test concurrency
performance(by running the existing JUnit tests).
I started it because I found that I can't do something as I want with JUnit,
:) like concurrency, performance test. Why do I have to start thread, and
join, and check the result at the end of join everytime? why do I need to
use System.currentMilliseconds and sysout... :) So I wrote p-unit.
3. Running tests in several VM is great, but how could I compare these
> results?
There're serveral build-in reports in p-unit. The most straightforward one
is pdf format, which contains performance chart. The results of different vm
for the same method are shown in the same page, so that you can know which
one is better easily. It's also easy to plugin your own report by
implementing PUnitEventListener.
4. If I need to increase a parameter until the test fails and then
> report the final value as a test performance metrics how this should
> be done in p-unit?
hm... this functionality is not supported in p-unit....
Thanks!
On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> > On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> > >
> > > Hello Vladimir,
> > >
> > > Could yout please give few hints how one should write performance
> > > tests using the harness? I found an option "-performance" but cannot
> > > see anything else. I wonder how a testcase could report a performance
> > > metrics.
> > >
> > > Actually I'm thinking now of reviving comparative stress testing
> approach.
> >
> >
> > Hi Alexei,
> >
> > Do you have a chance to look at p-unit, (http://p-unit.sourceforge.net/
> )?
> >
> > I think it fulfills most of the requests listed in the documents.
> >
> > Another advantage of p-unit is that it can compare the performance
> between
> > different vms.
> >
> > One problem may be the license (GPL now, going to change the license to
> GPL
> > v3). If we really take it serious, I can relicense it immediately.
> >
> > [1]
> > >
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew Zhang
> >
> > http://zhanghuangzhu.blogspot.com/
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/
Re: [buildtest][harness] performance measurement infrastructure
Posted by Alexei Fedotov <al...@gmail.com>.
Andrew,
This is very interesting. Thank you for the reference.
I've got few questions trying to understand your framework.
1. Why an interface Watcher extends Serializable?
2. Is there any project which uses p-unit for testing? Why have you started it?
3. Running tests in several VM is great, but how could I compare these results?
4. If I need to increase a parameter until the test fails and then
report the final value as a test performance metrics how this should
be done in p-unit?
On 6/22/07, Andrew Zhang <zh...@gmail.com> wrote:
> On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
> >
> > Hello Vladimir,
> >
> > Could yout please give few hints how one should write performance
> > tests using the harness? I found an option "-performance" but cannot
> > see anything else. I wonder how a testcase could report a performance
> > metrics.
> >
> > Actually I'm thinking now of reviving comparative stress testing approach.
>
>
> Hi Alexei,
>
> Do you have a chance to look at p-unit, (http://p-unit.sourceforge.net/)?
>
> I think it fulfills most of the requests listed in the documents.
>
> Another advantage of p-unit is that it can compare the performance between
> different vms.
>
> One problem may be the license (GPL now, going to change the license to GPL
> v3). If we really take it serious, I can relicense it immediately.
>
> [1]
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest][harness] performance measurement infrastructure
Posted by Andrew Zhang <zh...@gmail.com>.
On 6/21/07, Alexei Fedotov <al...@gmail.com> wrote:
>
> Hello Vladimir,
>
> Could yout please give few hints how one should write performance
> tests using the harness? I found an option "-performance" but cannot
> see anything else. I wonder how a testcase could report a performance
> metrics.
>
> Actually I'm thinking now of reviving comparative stress testing approach.
Hi Alexei,
Do you have a chance to look at p-unit, (http://p-unit.sourceforge.net/)?
I think it fulfills most of the requests listed in the documents.
Another advantage of p-unit is that it can compare the performance between
different vms.
One problem may be the license (GPL now, going to change the license to GPL
v3). If we really take it serious, I can relicense it immediately.
[1]
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Best regards,
Andrew Zhang
http://zhanghuangzhu.blogspot.com/