You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by "Jimmy, Jing Lv" <fi...@gmail.com> on 2006/08/15 12:34:25 UTC
[classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument]
Documents for non-unit-tests)
Hi,
I'm trying to write some unit tests of instrument. As discussed,
I'm using exec.
Currently I've tried Support_Exec and find it fairly easy to
use(Thanks Stepan :) ).
For resources (jar files and utility classes), I make a new directory:
$Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
and put them there.
And I meet a problem here. Currently the default classpath in
running test is "$Harmony/modules/instrument/bin", so it meets problem
in loading those utility classes (it can load jar files though).
I remember Mark has plan to change the classpath in running
testcase, it'll be great if he make some progress.
An another problem is checking, currently the result is checked by
searching and compare in string(e.g, find "someException" in result).
But the framework seems fails if any exception is thrown out. Can it
make a little change that it passes and returns the error message in exec?
Jimmy, Jing Lv wrote:
> Stepan Mishura wrote:
>> On 8/7/06, *Jimmy, Jing Lv* wrote:
>>
>> <SNIP>
>> Sounds pretty good :)
>> "exec" do helps, it can check simple situations.
>> What I'm concern is that if the return code is not enough for some
>> situations, e.g, what exception is thrown exactly, or what cause
>> VM exit
>> abnormally. IMHO, it is still necessary for us.
>>
>>
>> Hi Jimmy,
>>
>> See attached test. It demonstrates VM forking. The test saves VM
>> output and prints it in the end.
>> So I expect that unit tests for 'instrument' module can do the same:
>> fork VM with predefined args, save output and compare it with expected
>> output.
>>
>> BWT, we have utility class[1] in 'support' for this task. It makes
>> sense to improve it for our needs.
>>
>> Thanks,
>> Stepan.
>>
>
> Thanks Stepan, this is very attractive to me :)
>
> According to your approach, I believe now it can be automatized well, at
> least for many test.
>
> One concern from me is that, how do you handle output comparison? For an
> example, if VM halt with a exception, it output some message, so you get
> the output, find the exception name in it? And what can we do if it
> output some information only, which RI and Harmony may not appear the same?
>
> However this is a good idea, if no objections, I shall follow this
> approach in writing some test of instrument.
>
>> [1]
>> http://svn.apache.org/viewvc/incubator/harmony/enhanced/classlib/trunk/support/src/test/java/tests/support/Support_Exec.java?revision=386058&view=markup
>> <http://svn.apache.org/viewvc/incubator/harmony/enhanced/classlib/trunk/support/src/test/java/tests/support/Support_Exec.java?revision=386058&view=markup>
>>
--
Best Regards!
Jimmy, Jing Lv
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Gregory Shimansky <gs...@gmail.com>.
On Wednesday 16 August 2006 16:13 Anton Luht wrote:
> > For example, how you will test that VM is aborted during start-up with
> > NoClassDefFoundError (not with IllegalArgumentException or any other
> > exception) if it is impossible to load requested class?
>
> I think it should be done using some VM tests - maybe using JVMTI.
> Parsing output sounds too weak for me - anything that prints
> "NoClassDefFoundError args[0]" will do.
>
> This is too complicated though and I agree that parsing error output
> will work in most cases.
I've lost the beginning of the discussion so I've considered a limited number
of the emails sent today only. If I understood correctly you want to test a
correct exception for a child VM process and consider using JVMTI as the
testing solution for such exception.
I want to point out that JVMTI spec is quite vague in the description of VM
startup events sequence, the only guaranteed sequence statement can be found
in [1]. The LIVE phase should begin after the main thread has been started.
It says nothing about ClassLoad event for main application class, it may be
loaded before the main thread is created, therefore before LIVE phase.
The Exception event can and should occur only in LIVE phase, that is after
VMInit event has been sent. VM can fail and never send the VMInit event if it
can't find main application class or some class like the one supplied
via -javaagent option before LIVE phase has begun.
So I think this testing approach may fail if it relies on unspecified VM
behavior.
[1] http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html#VMInit
--
Gregory Shimansky, Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Anton Luht <an...@gmail.com>.
> For example, how you will test that VM is aborted during start-up with
> NoClassDefFoundError (not with IllegalArgumentException or any other
> exception) if it is impossible to load requested class?
I think it should be done using some VM tests - maybe using JVMTI.
Parsing output sounds too weak for me - anything that prints
"NoClassDefFoundError args[0]" will do.
This is too complicated though and I agree that parsing error output
will work in most cases.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mbox/%3c44D6AC2E.4000603@gmail.com%3e
>
> On 8/16/06, Stepan Mishura <st...@gmail.com> wrote:
> > > On 8/16/06, Anton Luht wrote:
> > > >
> > > > Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
> > > > dies because some exception (say, NPE) not catched, support_exec
> > > > analyzes output, understands that NPE was thrown and rethrows it :)
> > > > People who will look at those tests will certainly go mad.
> > >
> > >
> > > You misunderstoond the idea for using support_exec - no exceptions
> > > re-throwing should be made. A test will verify only that VM fails in
> > > expected way (i.e. with expected error message).
> > >
> > > Do you have a better idea how to test that?
> > >
> > > Thanks,
> > > Stepan.
> > >
> > > On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
> > > > > Paulex Yang wrote:
> > > > > > Jimmy, Jing Lv wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> I'm trying to write some unit tests of instrument. As
> > discussed,
> > > > > >> I'm using exec.
> > > > > >> Currently I've tried Support_Exec and find it fairly easy to
> > > > > >> use(Thanks Stepan :) ).
> > > > > >>
> > > > > >> For resources (jar files and utility classes), I make a new
> > > > > >> directory:
> > > > > >>
> > > >
> > $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> > > > > >> and put them there.
> > > > > >>
> > > > > >> And I meet a problem here. Currently the default classpath in
> > > > > >> running test is "$Harmony/modules/instrument/bin", so it meets
> > > > problem
> > > > > >> in loading those utility classes (it can load jar files though).
> > > > > >> I remember Mark has plan to change the classpath in running
> > > > > >> testcase, it'll be great if he make some progress.
> > > > > >>
> > > > > >> An another problem is checking, currently the result is
> > checked
> > > > by
> > > > > >> searching and compare in string(e.g, find "someException" in
> > result).
> > > > > >> But the framework seems fails if any exception is thrown out. Can
> > it
> > > > > >> make a little change that it passes and returns the error message
> > in
> > > > > >> exec?
> > > > > > Didn't catch up here, did you mean that you need to check the
> > > > exception
> > > > > > message as well as exception type?
> > > > >
> > > > > Not necessary to check messages. However we can not catch those
> > > > > exceptions so we have to analysis output. For an example:
> > > > >
> > > > > run "java -javaagent:test.jar myclass"
> > > > >
> > > > > so get output:
> > > > > Exception in thread "main"
> > java.lang.reflect.InvocationTargetException
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > Source)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > Source)
> > > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > at
> > > > > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown
> > So
> > > > > urce)
> > > > > Caused by: java.lang.IllegalAccessError
> > > > > ...
> > > > >
> > > > > In this case we can search for "InvocationTargetException" in the
> > output
> > > > > to ensure the expected exception. Sometimes we may also check "cause
> > by"
> > > > > exceptions.
> > > > >
> > > > > This approach seems not perfect. Any comments/suggestions? Thanks!
> > > > >
> > > > > <snip>
> > > > > --
> > > > >
> > > > > Best Regards!
> > > > >
> > > > > Jimmy, Jing Lv
> > > > > China Software Development Lab, IBM
> >
> > ------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Regards,
Anton Luht,
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Leo Li <li...@gmail.com>.
Hi, all:
I wonder if we need to get the exception from the exec framework. As an
exec result, we need only get whether it succeeds or not. Besides, I do not
think it is so reliable to parse error message to get exception.
We can design sophisticated testcase to deduce whether the expected
exception has been thrown out from the status of exec.
For example, during VM init there are some errors that can be caught by
some java class.
If we add a null object as Transformer, the VM will go down by
nullpointerexception. But it can be caught be its caller:
public static void premain(String agentArgs, Instrumentation inst)
{
try
{
inst.addTransformer(null);
//should throw NullPointerException
System.exit(1);
}
catch(NullPointerException e)
{
//expected
}
}
From the exec result, if there is no error, that is return value is 0, we
can deduce that the expected exception is thrown.
As for those fault errors raised from VM init, if they cannot be caught
by any java class, they are just error messages that can be got from
console. I do not think some application will depend on such message.
Furthermore, some of these messages cannot be classified to any exception
type: some are errors about runtime environment, some are related to VM.
Actually, in reality, some error events are related to Harmony's
implemenation. For example, in our instrumentation impl class, there are two
methods executePremain and transform, which are called duirng VM and might
cause VM to die if these methods cannot be invoked. In such circumstances,
we cannot behave just like RI.
So in my opinion, if the exception can be caught by any java class, we try
to design testcase to verify it; or else, they are just error messages that
are related to implementation.
On 8/17/06, Anton Luht <an...@gmail.com> wrote:
>
> Jimmy,
>
> > Ah, we are trying to catch the exact exception name, not its message. I
> > agree parsing the message is a bad idea, but find an exact exception
> > name in the output is still stable, am I right?
>
> Yes, it's stable.
>
> > In fact, I have no idea to catch an exception in "exec", it surely
> > contains no such function, right?
>
> No, it doesn't.
>
> --
> Regards,
> Anton Luht,
> Intel Middleware Products Division
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Leo Li
China Software Development Lab, IBM
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Anton Luht <an...@gmail.com>.
Jimmy,
> Ah, we are trying to catch the exact exception name, not its message. I
> agree parsing the message is a bad idea, but find an exact exception
> name in the output is still stable, am I right?
Yes, it's stable.
> In fact, I have no idea to catch an exception in "exec", it surely
> contains no such function, right?
No, it doesn't.
--
Regards,
Anton Luht,
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument]
Documents for non-unit-tests)
Posted by "Jimmy, Jing Lv" <fi...@gmail.com>.
Anton Luht wrote:
> IMHO analysis of exception stacktrace output is a bad idea. Tests
> should not produce uncatched exceptions unless they fail. Test authors
> should explicitly catch an exception and print some text to stdout.
> Reasons are:
>
> - exception messages are not standard, different JVMs can produce
> different output
> - localized messages can be printed, the test may fail because of locales
> - if test author expects IOException but subclass of this exception is
> thrown and printed, the test will fail while if the test was written
> in the way "catch IOException; print "PASSED" the test would pass
>
>
Ah, we are trying to catch the exact exception name, not its message. I
agree parsing the message is a bad idea, but find an exact exception
name in the output is still stable, am I right?
In fact, I have no idea to catch an exception in "exec", it surely
contains no such function, right?
> On 8/16/06, Stepan Mishura <st...@gmail.com> wrote:
>> On 8/16/06, Anton Luht wrote:
>> >
>> > Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
>> > dies because some exception (say, NPE) not catched, support_exec
>> > analyzes output, understands that NPE was thrown and rethrows it :)
>> > People who will look at those tests will certainly go mad.
>>
>>
>> You misunderstoond the idea for using support_exec - no exceptions
>> re-throwing should be made. A test will verify only that VM fails in
>> expected way (i.e. with expected error message).
>>
>> Do you have a better idea how to test that?
>>
>> Thanks,
>> Stepan.
>>
>> On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
>> > > Paulex Yang wrote:
>> > > > Jimmy, Jing Lv wrote:
>> > > >> Hi,
>> > > >>
>> > > >> I'm trying to write some unit tests of instrument. As
>> discussed,
>> > > >> I'm using exec.
>> > > >> Currently I've tried Support_Exec and find it fairly easy to
>> > > >> use(Thanks Stepan :) ).
>> > > >>
>> > > >> For resources (jar files and utility classes), I make a new
>> > > >> directory:
>> > > >>
>> >
>> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
>>
>> > > >> and put them there.
>> > > >>
>> > > >> And I meet a problem here. Currently the default classpath in
>> > > >> running test is "$Harmony/modules/instrument/bin", so it meets
>> > problem
>> > > >> in loading those utility classes (it can load jar files though).
>> > > >> I remember Mark has plan to change the classpath in running
>> > > >> testcase, it'll be great if he make some progress.
>> > > >>
>> > > >> An another problem is checking, currently the result is
>> checked
>> > by
>> > > >> searching and compare in string(e.g, find "someException" in
>> result).
>> > > >> But the framework seems fails if any exception is thrown out.
>> Can it
>> > > >> make a little change that it passes and returns the error
>> message in
>> > > >> exec?
>> > > > Didn't catch up here, did you mean that you need to check the
>> > exception
>> > > > message as well as exception type?
>> > >
>> > > Not necessary to check messages. However we can not catch those
>> > > exceptions so we have to analysis output. For an example:
>> > >
>> > > run "java -javaagent:test.jar myclass"
>> > >
>> > > so get output:
>> > > Exception in thread "main"
>> java.lang.reflect.InvocationTargetException
>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
>> Source)
>> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>> > Source)
>> > > at java.lang.reflect.Method.invoke(Unknown Source)
>> > > at
>> > > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So
>> > > urce)
>> > > Caused by: java.lang.IllegalAccessError
>> > > ...
>> > >
>> > > In this case we can search for "InvocationTargetException" in the
>> output
>> > > to ensure the expected exception. Sometimes we may also check
>> "cause by"
>> > > exceptions.
>> > >
>> > > This approach seems not perfect. Any comments/suggestions? Thanks!
>> > >
>> > > <snip>
>> > > --
>> > >
>> > > Best Regards!
>> > >
>> > > Jimmy, Jing Lv
>> > > China Software Development Lab, IBM
>> > >
>> >
>> >
>>
>>
>> --
>> Thanks,
>> Stepan Mishura
>> Intel Middleware Products Division
>>
>> ------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>
--
Best Regards!
Jimmy, Jing Lv
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Stepan Mishura <st...@gmail.com>.
On 8/16/06, Anton Luht wrote:
>
> IMHO analysis of exception stacktrace output is a bad idea. Tests
> should not produce uncatched exceptions unless they fail. Test authors
> should explicitly catch an exception and print some text to stdout.
See [1] for testing details.
Reasons are:
>
> - exception messages are not standard, different JVMs can produce
> different output
> - localized messages can be printed, the test may fail because of locales
> - if test author expects IOException but subclass of this exception is
> thrown and printed, the test will fail while if the test was written
> in the way "catch IOException; print "PASSED" the test would pass
For example, how you will test that VM is aborted during start-up with
NoClassDefFoundError (not with IllegalArgumentException or any other
exception) if it is impossible to load requested class?
Thanks,
Stepan.
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mbox/%3c44D6AC2E.4000603@gmail.com%3e
On 8/16/06, Stepan Mishura <st...@gmail.com> wrote:
> > On 8/16/06, Anton Luht wrote:
> > >
> > > Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
> > > dies because some exception (say, NPE) not catched, support_exec
> > > analyzes output, understands that NPE was thrown and rethrows it :)
> > > People who will look at those tests will certainly go mad.
> >
> >
> > You misunderstoond the idea for using support_exec - no exceptions
> > re-throwing should be made. A test will verify only that VM fails in
> > expected way (i.e. with expected error message).
> >
> > Do you have a better idea how to test that?
> >
> > Thanks,
> > Stepan.
> >
> > On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
> > > > Paulex Yang wrote:
> > > > > Jimmy, Jing Lv wrote:
> > > > >> Hi,
> > > > >>
> > > > >> I'm trying to write some unit tests of instrument. As
> discussed,
> > > > >> I'm using exec.
> > > > >> Currently I've tried Support_Exec and find it fairly easy to
> > > > >> use(Thanks Stepan :) ).
> > > > >>
> > > > >> For resources (jar files and utility classes), I make a new
> > > > >> directory:
> > > > >>
> > >
> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> > > > >> and put them there.
> > > > >>
> > > > >> And I meet a problem here. Currently the default classpath in
> > > > >> running test is "$Harmony/modules/instrument/bin", so it meets
> > > problem
> > > > >> in loading those utility classes (it can load jar files though).
> > > > >> I remember Mark has plan to change the classpath in running
> > > > >> testcase, it'll be great if he make some progress.
> > > > >>
> > > > >> An another problem is checking, currently the result is
> checked
> > > by
> > > > >> searching and compare in string(e.g, find "someException" in
> result).
> > > > >> But the framework seems fails if any exception is thrown out. Can
> it
> > > > >> make a little change that it passes and returns the error message
> in
> > > > >> exec?
> > > > > Didn't catch up here, did you mean that you need to check the
> > > exception
> > > > > message as well as exception type?
> > > >
> > > > Not necessary to check messages. However we can not catch those
> > > > exceptions so we have to analysis output. For an example:
> > > >
> > > > run "java -javaagent:test.jar myclass"
> > > >
> > > > so get output:
> > > > Exception in thread "main"
> java.lang.reflect.InvocationTargetException
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source)
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > at
> > > > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown
> So
> > > > urce)
> > > > Caused by: java.lang.IllegalAccessError
> > > > ...
> > > >
> > > > In this case we can search for "InvocationTargetException" in the
> output
> > > > to ensure the expected exception. Sometimes we may also check "cause
> by"
> > > > exceptions.
> > > >
> > > > This approach seems not perfect. Any comments/suggestions? Thanks!
> > > >
> > > > <snip>
> > > > --
> > > >
> > > > Best Regards!
> > > >
> > > > Jimmy, Jing Lv
> > > > China Software Development Lab, IBM
>
> ------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Anton Luht <an...@gmail.com>.
IMHO analysis of exception stacktrace output is a bad idea. Tests
should not produce uncatched exceptions unless they fail. Test authors
should explicitly catch an exception and print some text to stdout.
Reasons are:
- exception messages are not standard, different JVMs can produce
different output
- localized messages can be printed, the test may fail because of locales
- if test author expects IOException but subclass of this exception is
thrown and printed, the test will fail while if the test was written
in the way "catch IOException; print "PASSED" the test would pass
On 8/16/06, Stepan Mishura <st...@gmail.com> wrote:
> On 8/16/06, Anton Luht wrote:
> >
> > Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
> > dies because some exception (say, NPE) not catched, support_exec
> > analyzes output, understands that NPE was thrown and rethrows it :)
> > People who will look at those tests will certainly go mad.
>
>
> You misunderstoond the idea for using support_exec - no exceptions
> re-throwing should be made. A test will verify only that VM fails in
> expected way (i.e. with expected error message).
>
> Do you have a better idea how to test that?
>
> Thanks,
> Stepan.
>
> On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
> > > Paulex Yang wrote:
> > > > Jimmy, Jing Lv wrote:
> > > >> Hi,
> > > >>
> > > >> I'm trying to write some unit tests of instrument. As discussed,
> > > >> I'm using exec.
> > > >> Currently I've tried Support_Exec and find it fairly easy to
> > > >> use(Thanks Stepan :) ).
> > > >>
> > > >> For resources (jar files and utility classes), I make a new
> > > >> directory:
> > > >>
> > $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> > > >> and put them there.
> > > >>
> > > >> And I meet a problem here. Currently the default classpath in
> > > >> running test is "$Harmony/modules/instrument/bin", so it meets
> > problem
> > > >> in loading those utility classes (it can load jar files though).
> > > >> I remember Mark has plan to change the classpath in running
> > > >> testcase, it'll be great if he make some progress.
> > > >>
> > > >> An another problem is checking, currently the result is checked
> > by
> > > >> searching and compare in string(e.g, find "someException" in result).
> > > >> But the framework seems fails if any exception is thrown out. Can it
> > > >> make a little change that it passes and returns the error message in
> > > >> exec?
> > > > Didn't catch up here, did you mean that you need to check the
> > exception
> > > > message as well as exception type?
> > >
> > > Not necessary to check messages. However we can not catch those
> > > exceptions so we have to analysis output. For an example:
> > >
> > > run "java -javaagent:test.jar myclass"
> > >
> > > so get output:
> > > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So
> > > urce)
> > > Caused by: java.lang.IllegalAccessError
> > > ...
> > >
> > > In this case we can search for "InvocationTargetException" in the output
> > > to ensure the expected exception. Sometimes we may also check "cause by"
> > > exceptions.
> > >
> > > This approach seems not perfect. Any comments/suggestions? Thanks!
> > >
> > > <snip>
> > > --
> > >
> > > Best Regards!
> > >
> > > Jimmy, Jing Lv
> > > China Software Development Lab, IBM
> > >
> >
> >
>
>
> --
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
>
> ------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Regards,
Anton Luht,
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Stepan Mishura <st...@gmail.com>.
On 8/16/06, Anton Luht wrote:
>
> Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
> dies because some exception (say, NPE) not catched, support_exec
> analyzes output, understands that NPE was thrown and rethrows it :)
> People who will look at those tests will certainly go mad.
You misunderstoond the idea for using support_exec - no exceptions
re-throwing should be made. A test will verify only that VM fails in
expected way (i.e. with expected error message).
Do you have a better idea how to test that?
Thanks,
Stepan.
On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
> > Paulex Yang wrote:
> > > Jimmy, Jing Lv wrote:
> > >> Hi,
> > >>
> > >> I'm trying to write some unit tests of instrument. As discussed,
> > >> I'm using exec.
> > >> Currently I've tried Support_Exec and find it fairly easy to
> > >> use(Thanks Stepan :) ).
> > >>
> > >> For resources (jar files and utility classes), I make a new
> > >> directory:
> > >>
> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> > >> and put them there.
> > >>
> > >> And I meet a problem here. Currently the default classpath in
> > >> running test is "$Harmony/modules/instrument/bin", so it meets
> problem
> > >> in loading those utility classes (it can load jar files though).
> > >> I remember Mark has plan to change the classpath in running
> > >> testcase, it'll be great if he make some progress.
> > >>
> > >> An another problem is checking, currently the result is checked
> by
> > >> searching and compare in string(e.g, find "someException" in result).
> > >> But the framework seems fails if any exception is thrown out. Can it
> > >> make a little change that it passes and returns the error message in
> > >> exec?
> > > Didn't catch up here, did you mean that you need to check the
> exception
> > > message as well as exception type?
> >
> > Not necessary to check messages. However we can not catch those
> > exceptions so we have to analysis output. For an example:
> >
> > run "java -javaagent:test.jar myclass"
> >
> > so get output:
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So
> > urce)
> > Caused by: java.lang.IllegalAccessError
> > ...
> >
> > In this case we can search for "InvocationTargetException" in the output
> > to ensure the expected exception. Sometimes we may also check "cause by"
> > exceptions.
> >
> > This approach seems not perfect. Any comments/suggestions? Thanks!
> >
> > <snip>
> > --
> >
> > Best Regards!
> >
> > Jimmy, Jing Lv
> > China Software Development Lab, IBM
> >
>
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument] Documents for non-unit-tests)
Posted by Anton Luht <an...@gmail.com>.
Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that
dies because some exception (say, NPE) not catched, support_exec
analyzes output, understands that NPE was thrown and rethrows it :)
People who will look at those tests will certainly go mad.
On 8/16/06, Jimmy, Jing Lv <fi...@gmail.com> wrote:
> Paulex Yang wrote:
> > Jimmy, Jing Lv wrote:
> >> Hi,
> >>
> >> I'm trying to write some unit tests of instrument. As discussed,
> >> I'm using exec.
> >> Currently I've tried Support_Exec and find it fairly easy to
> >> use(Thanks Stepan :) ).
> >>
> >> For resources (jar files and utility classes), I make a new
> >> directory:
> >> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> >> and put them there.
> >>
> >> And I meet a problem here. Currently the default classpath in
> >> running test is "$Harmony/modules/instrument/bin", so it meets problem
> >> in loading those utility classes (it can load jar files though).
> >> I remember Mark has plan to change the classpath in running
> >> testcase, it'll be great if he make some progress.
> >>
> >> An another problem is checking, currently the result is checked by
> >> searching and compare in string(e.g, find "someException" in result).
> >> But the framework seems fails if any exception is thrown out. Can it
> >> make a little change that it passes and returns the error message in
> >> exec?
> > Didn't catch up here, did you mean that you need to check the exception
> > message as well as exception type?
>
> Not necessary to check messages. However we can not catch those
> exceptions so we have to analysis output. For an example:
>
> run "java -javaagent:test.jar myclass"
>
> so get output:
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So
> urce)
> Caused by: java.lang.IllegalAccessError
> ...
>
> In this case we can search for "InvocationTargetException" in the output
> to ensure the expected exception. Sometimes we may also check "cause by"
> exceptions.
>
> This approach seems not perfect. Any comments/suggestions? Thanks!
>
> <snip>
> --
>
> Best Regards!
>
> Jimmy, Jing Lv
> China Software Development Lab, IBM
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Regards,
Anton Luht,
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument]
Documents for non-unit-tests)
Posted by "Jimmy, Jing Lv" <fi...@gmail.com>.
Paulex Yang wrote:
> Jimmy, Jing Lv wrote:
>> Hi,
>>
>> I'm trying to write some unit tests of instrument. As discussed,
>> I'm using exec.
>> Currently I've tried Support_Exec and find it fairly easy to
>> use(Thanks Stepan :) ).
>>
>> For resources (jar files and utility classes), I make a new
>> directory:
>> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
>> and put them there.
>>
>> And I meet a problem here. Currently the default classpath in
>> running test is "$Harmony/modules/instrument/bin", so it meets problem
>> in loading those utility classes (it can load jar files though).
>> I remember Mark has plan to change the classpath in running
>> testcase, it'll be great if he make some progress.
>>
>> An another problem is checking, currently the result is checked by
>> searching and compare in string(e.g, find "someException" in result).
>> But the framework seems fails if any exception is thrown out. Can it
>> make a little change that it passes and returns the error message in
>> exec?
> Didn't catch up here, did you mean that you need to check the exception
> message as well as exception type?
Not necessary to check messages. However we can not catch those
exceptions so we have to analysis output. For an example:
run "java -javaagent:test.jar myclass"
so get output:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So
urce)
Caused by: java.lang.IllegalAccessError
...
In this case we can search for "InvocationTargetException" in the output
to ensure the expected exception. Sometimes we may also check "cause by"
exceptions.
This approach seems not perfect. Any comments/suggestions? Thanks!
<snip>
--
Best Regards!
Jimmy, Jing Lv
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][instrument]Using Support_Exec framework(was Re: [classlib][Instrument]
Documents for non-unit-tests)
Posted by Paulex Yang <pa...@gmail.com>.
Jimmy, Jing Lv wrote:
> Hi,
>
> I'm trying to write some unit tests of instrument. As discussed,
> I'm using exec.
> Currently I've tried Support_Exec and find it fairly easy to
> use(Thanks Stepan :) ).
>
> For resources (jar files and utility classes), I make a new
> directory:
> $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument
> and put them there.
>
> And I meet a problem here. Currently the default classpath in
> running test is "$Harmony/modules/instrument/bin", so it meets problem
> in loading those utility classes (it can load jar files though).
> I remember Mark has plan to change the classpath in running
> testcase, it'll be great if he make some progress.
>
> An another problem is checking, currently the result is checked by
> searching and compare in string(e.g, find "someException" in result).
> But the framework seems fails if any exception is thrown out. Can it
> make a little change that it passes and returns the error message in
> exec?
Didn't catch up here, did you mean that you need to check the exception
message as well as exception type?
>
>
> Jimmy, Jing Lv wrote:
>> Stepan Mishura wrote:
>>> On 8/7/06, *Jimmy, Jing Lv* wrote:
>>>
>>> <SNIP>
>>> Sounds pretty good :)
>>> "exec" do helps, it can check simple situations.
>>> What I'm concern is that if the return code is not enough for some
>>> situations, e.g, what exception is thrown exactly, or what cause
>>> VM exit
>>> abnormally. IMHO, it is still necessary for us.
>>>
>>>
>>> Hi Jimmy,
>>>
>>> See attached test. It demonstrates VM forking. The test saves VM
>>> output and prints it in the end.
>>> So I expect that unit tests for 'instrument' module can do the same:
>>> fork VM with predefined args, save output and compare it with
>>> expected output.
>>>
>>> BWT, we have utility class[1] in 'support' for this task. It makes
>>> sense to improve it for our needs.
>>>
>>> Thanks,
>>> Stepan.
>>>
>>
>> Thanks Stepan, this is very attractive to me :)
>>
>> According to your approach, I believe now it can be automatized well,
>> at least for many test.
>>
>> One concern from me is that, how do you handle output comparison? For
>> an example, if VM halt with a exception, it output some message, so
>> you get the output, find the exception name in it? And what can we do
>> if it output some information only, which RI and Harmony may not
>> appear the same?
>>
>> However this is a good idea, if no objections, I shall follow this
>> approach in writing some test of instrument.
>>
>>> [1]
>>> http://svn.apache.org/viewvc/incubator/harmony/enhanced/classlib/trunk/support/src/test/java/tests/support/Support_Exec.java?revision=386058&view=markup
>>> <http://svn.apache.org/viewvc/incubator/harmony/enhanced/classlib/trunk/support/src/test/java/tests/support/Support_Exec.java?revision=386058&view=markup>
>>>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org