You are viewing a plain text version of this content. The canonical link for it is here.
Posted to kato-dev@incubator.apache.org by Stuart Monteith <st...@stoo.me.uk> on 2009/10/22 14:51:13 UTC
TCK and all that
Hi everybody,
Apologies for the cross posting - but the implementation of the TCK
is relevant to both camps.
One of the things we need to do before making a release available is to
produce a TCK. By "TCK" I mean
a test harness that will:
1. Setup a JVM into a known configuration and generate a dump.
2. Open a dump with the Kato API and run unit tests against it.
Whether or not these are just functional tests or technology compliance
is something I will just gloss over for now
as just a classification issue.
As we all know, there is a problem with optionality in the API. There is
information that will be lost when using
certain dump types.
Steve has suggested two modes for running the TCK.
Legacy
========
This mode runs through the API and confirms that the basic API behaves
as it is intended to do so.
It checks that the rules of the API are adhered to, but doesn't expect
any method will return any
particular piece of information.
For example:
List<ImageProcess> ImageAddressSpace.getProcesses();
processes would be checked to ensure that it is not null. If there were
contents
they would be queried, and their methods executed.
For example:
String ImageProcess.getID();
Would be expected to return a String or throw a DataUnavailable or
CorruptDataException exception.
The actual contents of the string wouldn't be checked - just so long as
it isn't null.
(incidentally, actually checking the value in this example for
correctness is difficult).
1.0
=====
This is a test for exact compliance with the API. All of the information
that can be retrieved from the dump
must be retrieved and it must all be correct.
With some caveats.
Like with ImageProcess.getID(), there is some information that is poorly
specified or difficult to know in
advance.
The poorly specified aspects of the API are:
Platform specific items.
For example:
Implementation specific items:
toString() methods - what do they print?
Things that are difficult to know in advance:
Native stack frames - ordering and naming.
JavaThreads - there will be threads that are implementation specific.
There problems aren't intractable. A certain amount of flexibility will
be necessary to accommodate variations.
We should expect there to be more than what we expect, so we should just
look for what we expect to be there,
and ignore the rest.
The poorly specified aspects of the API should just be more precisely
specified. Although it means that the TCK will
have to be aware of the different platforms and their behaviour.
I'm not a great fan of the "Legacy" mode. While it may have a basic use for testing basic behaviour, I would like for there to be stronger statement about what we expect the API to do.
For example, if we call JavaRuntime.getThreads(), I would expect a List<JavaThread> to be returned. However, if it wasn't empty, I would want the threads we put into the testcase to be there and identifiable by their (correct) names.
i.e. if there is something to test we should test it.
I call this mode "Permissive" (alternatives: lenient, acquiescent, lax, liberal, tolerant)
There are, of course, problems. But I would expect that is this mode wasn't possible or advisable, it would be improbable that a program could be writtent to call the API.
The end result is that if you get information out of the API, it must be correct. Where you don't, it is ignored.
Thoughts?
Thoughts?
Regards,
Stuart
--
Stuart Monteith
http://blog.stoo.me.uk/
RE: TCK and all that
Posted by "Bobrovsky, Konstantin S" <ko...@intel.com>.
> (2) use RI to read the snapshot and retrieve necessary information, then
Not 'RI', but 'KATO implementation to be tested', of course. Sorry for confusion.
Thanks,
Konst
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
>-----Original Message-----
>From: Bobrovsky, Konstantin S [mailto:konstantin.s.bobrovsky@intel.com]
>Sent: Thursday, October 29, 2009 5:28 PM
>To: kato-spec@incubator.apache.org; kato-dev@incubator.apache.org
>Subject: RE: TCK and all that
>
>Hi Stuart,
>
>one of the approaches to write TCK tests for the "1.0" mode I can think of
>is making the test application retrieve an log information to be checked
>(using standard Java API means) to some "golden" file, then expecting the
>TCK test obtain the same (or "similar") information from a "snapshot" using
>the RI API.
>
>In more details:
>There are several agents:
> - KATO implementation to be tested
> - Java Runtime which runs the tests and is coupled with the KATO
> implementation
> - test application (a set of classes run as a sample payload application)
> - TCK test, which can work in 2 modes:
> (1) run the test application, retrieve and log all the information
> to be checked in mode (2) to a "golden file" (should be stable
> between invokations), have the snapshot to be generated by some
> means.
> (2) use RI to read the snapshot and retrieve necessary information, then
> Compare what's retrieved with the golden file
>Each such test is run 2 times: in mode (1) to generate the golden file and
>the dump, and in mode (2) when actual API implementation verification
>happens.
>
>Of course, not entire API can be tested this way, but a fair part can. The
>main question is what kind of information can be obtained in mode (1),
>which can later be retrieved from the snapshot. What comes to mind is:
> - all alive threads running in the test application and their Java stacks
> using java.lang.Thread.getAllStackTraces. For threads with pre-defined
> names which the test recognizes as "special" ones (and which are
> created by the test application), some field values or other details
> can be queried
> - a subset of live objects created by the test application
> - ... many more if JVMTI or JMX is used as the information provider
> in mode (1)
>
>Thanks,
>Konst
>
>Intel Novosibirsk
>Closed Joint Stock Company Intel A/O
>Registered legal address: Krylatsky Hills Business Park,
>17 Krylatskaya Str., Bldg 4, Moscow 121614,
>Russian Federation
>
>
>>-----Original Message-----
>>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>>Sent: Thursday, October 22, 2009 9:51 PM
>>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>>Subject: TCK and all that
>>
>>Hi everybody,
>>
>> Apologies for the cross posting - but the implementation of the TCK
>>is relevant to both camps.
>>One of the things we need to do before making a release available is to
>>produce a TCK. By "TCK" I mean
>>a test harness that will:
>>
>>1. Setup a JVM into a known configuration and generate a dump.
>>2. Open a dump with the Kato API and run unit tests against it.
>>
>>Whether or not these are just functional tests or technology compliance
>>is something I will just gloss over for now
>>as just a classification issue.
>>
>>As we all know, there is a problem with optionality in the API. There is
>>information that will be lost when using
>>certain dump types.
>>
>>Steve has suggested two modes for running the TCK.
>>
>> Legacy
>>========
>>
>>This mode runs through the API and confirms that the basic API behaves
>>as it is intended to do so.
>>It checks that the rules of the API are adhered to, but doesn't expect
>>any method will return any
>>particular piece of information.
>>
>>For example:
>> List<ImageProcess> ImageAddressSpace.getProcesses();
>>
>>processes would be checked to ensure that it is not null. If there were
>>contents
>>they would be queried, and their methods executed.
>>
>>For example:
>> String ImageProcess.getID();
>>
>>Would be expected to return a String or throw a DataUnavailable or
>>CorruptDataException exception.
>>The actual contents of the string wouldn't be checked - just so long as
>>it isn't null.
>>(incidentally, actually checking the value in this example for
>>correctness is difficult).
>>
>> 1.0
>>=====
>>
>>This is a test for exact compliance with the API. All of the information
>>that can be retrieved from the dump
>>must be retrieved and it must all be correct.
>>
>>With some caveats.
>>
>>Like with ImageProcess.getID(), there is some information that is poorly
>>specified or difficult to know in
>>advance.
>>
>>The poorly specified aspects of the API are:
>> Platform specific items.
>> For example:
>>
>> Implementation specific items:
>> toString() methods - what do they print?
>>
>>Things that are difficult to know in advance:
>> Native stack frames - ordering and naming.
>> JavaThreads - there will be threads that are implementation specific.
>>
>>There problems aren't intractable. A certain amount of flexibility will
>>be necessary to accommodate variations.
>>We should expect there to be more than what we expect, so we should just
>>look for what we expect to be there,
>>and ignore the rest.
>>
>>The poorly specified aspects of the API should just be more precisely
>>specified. Although it means that the TCK will
>>have to be aware of the different platforms and their behaviour.
>>
>>
>>I'm not a great fan of the "Legacy" mode. While it may have a basic use
>for
>>testing basic behaviour, I would like for there to be stronger statement
>>about what we expect the API to do.
>>
>>For example, if we call JavaRuntime.getThreads(), I would expect a
>>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>>the threads we put into the testcase to be there and identifiable by their
>>(correct) names.
>>
>>i.e. if there is something to test we should test it.
>>
>>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>>liberal, tolerant)
>>
>>There are, of course, problems. But I would expect that is this mode
>wasn't
>>possible or advisable, it would be improbable that a program could be
>>writtent to call the API.
>>
>>The end result is that if you get information out of the API, it must be
>>correct. Where you don't, it is ignored.
>>
>>Thoughts?
>>
>>
>>Thoughts?
>>
>>Regards,
>> Stuart
>>
>>
>>
>>--
>>Stuart Monteith
>>http://blog.stoo.me.uk/
RE: TCK and all that
Posted by "Bobrovsky, Konstantin S" <ko...@intel.com>.
> (2) use RI to read the snapshot and retrieve necessary information, then
Not 'RI', but 'KATO implementation to be tested', of course. Sorry for confusion.
Thanks,
Konst
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
>-----Original Message-----
>From: Bobrovsky, Konstantin S [mailto:konstantin.s.bobrovsky@intel.com]
>Sent: Thursday, October 29, 2009 5:28 PM
>To: kato-spec@incubator.apache.org; kato-dev@incubator.apache.org
>Subject: RE: TCK and all that
>
>Hi Stuart,
>
>one of the approaches to write TCK tests for the "1.0" mode I can think of
>is making the test application retrieve an log information to be checked
>(using standard Java API means) to some "golden" file, then expecting the
>TCK test obtain the same (or "similar") information from a "snapshot" using
>the RI API.
>
>In more details:
>There are several agents:
> - KATO implementation to be tested
> - Java Runtime which runs the tests and is coupled with the KATO
> implementation
> - test application (a set of classes run as a sample payload application)
> - TCK test, which can work in 2 modes:
> (1) run the test application, retrieve and log all the information
> to be checked in mode (2) to a "golden file" (should be stable
> between invokations), have the snapshot to be generated by some
> means.
> (2) use RI to read the snapshot and retrieve necessary information, then
> Compare what's retrieved with the golden file
>Each such test is run 2 times: in mode (1) to generate the golden file and
>the dump, and in mode (2) when actual API implementation verification
>happens.
>
>Of course, not entire API can be tested this way, but a fair part can. The
>main question is what kind of information can be obtained in mode (1),
>which can later be retrieved from the snapshot. What comes to mind is:
> - all alive threads running in the test application and their Java stacks
> using java.lang.Thread.getAllStackTraces. For threads with pre-defined
> names which the test recognizes as "special" ones (and which are
> created by the test application), some field values or other details
> can be queried
> - a subset of live objects created by the test application
> - ... many more if JVMTI or JMX is used as the information provider
> in mode (1)
>
>Thanks,
>Konst
>
>Intel Novosibirsk
>Closed Joint Stock Company Intel A/O
>Registered legal address: Krylatsky Hills Business Park,
>17 Krylatskaya Str., Bldg 4, Moscow 121614,
>Russian Federation
>
>
>>-----Original Message-----
>>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>>Sent: Thursday, October 22, 2009 9:51 PM
>>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>>Subject: TCK and all that
>>
>>Hi everybody,
>>
>> Apologies for the cross posting - but the implementation of the TCK
>>is relevant to both camps.
>>One of the things we need to do before making a release available is to
>>produce a TCK. By "TCK" I mean
>>a test harness that will:
>>
>>1. Setup a JVM into a known configuration and generate a dump.
>>2. Open a dump with the Kato API and run unit tests against it.
>>
>>Whether or not these are just functional tests or technology compliance
>>is something I will just gloss over for now
>>as just a classification issue.
>>
>>As we all know, there is a problem with optionality in the API. There is
>>information that will be lost when using
>>certain dump types.
>>
>>Steve has suggested two modes for running the TCK.
>>
>> Legacy
>>========
>>
>>This mode runs through the API and confirms that the basic API behaves
>>as it is intended to do so.
>>It checks that the rules of the API are adhered to, but doesn't expect
>>any method will return any
>>particular piece of information.
>>
>>For example:
>> List<ImageProcess> ImageAddressSpace.getProcesses();
>>
>>processes would be checked to ensure that it is not null. If there were
>>contents
>>they would be queried, and their methods executed.
>>
>>For example:
>> String ImageProcess.getID();
>>
>>Would be expected to return a String or throw a DataUnavailable or
>>CorruptDataException exception.
>>The actual contents of the string wouldn't be checked - just so long as
>>it isn't null.
>>(incidentally, actually checking the value in this example for
>>correctness is difficult).
>>
>> 1.0
>>=====
>>
>>This is a test for exact compliance with the API. All of the information
>>that can be retrieved from the dump
>>must be retrieved and it must all be correct.
>>
>>With some caveats.
>>
>>Like with ImageProcess.getID(), there is some information that is poorly
>>specified or difficult to know in
>>advance.
>>
>>The poorly specified aspects of the API are:
>> Platform specific items.
>> For example:
>>
>> Implementation specific items:
>> toString() methods - what do they print?
>>
>>Things that are difficult to know in advance:
>> Native stack frames - ordering and naming.
>> JavaThreads - there will be threads that are implementation specific.
>>
>>There problems aren't intractable. A certain amount of flexibility will
>>be necessary to accommodate variations.
>>We should expect there to be more than what we expect, so we should just
>>look for what we expect to be there,
>>and ignore the rest.
>>
>>The poorly specified aspects of the API should just be more precisely
>>specified. Although it means that the TCK will
>>have to be aware of the different platforms and their behaviour.
>>
>>
>>I'm not a great fan of the "Legacy" mode. While it may have a basic use
>for
>>testing basic behaviour, I would like for there to be stronger statement
>>about what we expect the API to do.
>>
>>For example, if we call JavaRuntime.getThreads(), I would expect a
>>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>>the threads we put into the testcase to be there and identifiable by their
>>(correct) names.
>>
>>i.e. if there is something to test we should test it.
>>
>>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>>liberal, tolerant)
>>
>>There are, of course, problems. But I would expect that is this mode
>wasn't
>>possible or advisable, it would be improbable that a program could be
>>writtent to call the API.
>>
>>The end result is that if you get information out of the API, it must be
>>correct. Where you don't, it is ignored.
>>
>>Thoughts?
>>
>>
>>Thoughts?
>>
>>Regards,
>> Stuart
>>
>>
>>
>>--
>>Stuart Monteith
>>http://blog.stoo.me.uk/
Re: TCK and all that
Posted by Stuart Monteith <st...@stoo.me.uk>.
Hi Konstantin,
Thanks for your reply. The existing TCK works by running the
configuration
methods on JUnit test cases and then creating a dump. The JUnit test classes
are run again, only within the JUnit framework such that the test
methods are
executed this time. The advantage of this system is that the set up and
the checking
both exist within the same class file and so consistency is easily
maintained. Now,
one of the problems with that approach is that the both the setup and
the testcases have
to be compiled and run using at least Java 5.0, and yet we wish to
support Java 1.4.2.
The other problem is that what we look for is what we expect from our
program's configuration
of the JVM, and no more, so your suggestion for the JVM to introspect
itself while running
does potentially produce a more complete test of the API's abilities.
The only frustration
is with 1.4.2, which doesn't provide support for
Thread.getAllStackTraces(), JVMTI, and no doubt
other things. I think that is something that should be up for discussion.
Using JVMTI to store a file with the JVM state is an interesting
suggestion. If we take the JVM
state and write it out to a file with a JVMTI agent, what we end up
producing might end up being
the CJVMTI implementation of Kato that we have just now, which does
raise the question of how we
go about testing that. It is interesting that would could use the JVMTI
agent to verify alternative
implementations of the API.
Introspecting within Java could do:
o Thread.getAllStackTraces()
as you suggested. Provided the threads we are interested in are
parked, there should be some
consistency, although the time between calling
getAllStackTraces() and producing a dump could
produce inconsistencies, which might be just the thread calling
getAllStackTraces().
Cross-referencing class names in the stack-traces with their
class instances may be very difficult,
unless uniqueness can be guaranteed. I think JVMTI would be
better suited to this.
o Reflection
We could retrieve parts of the heap and class hierarchy using
reflection. However, private fields would be
problematic.
o java.lang.ClassLoader
The problem with this class is that it doesn't tell you what it
loaded, and it can only
tell you about the parent loader or the system classloader. So it
doesn't really allow complete
knowledge of the system.
There are certain things we can do within Java for the Image API:
o Image.getHostName(), Image.getIPAddreses()
could be easily fulfilled with the Java API.
o Image.getSystemType(),Image.getSystemSubType()
could be fulfilled using the System properties (e.g. os.name,
os.arch, os.version)
In addition, the files on the system could be read to work out
what system it is.
o Image.getInstalledMemory()
With knowledge of the OS, this might be available through shell
commands or available
in a special file. (e.g. /proc/meminfo)
o Image.getProcessorCount(), getProcessorType(), getProcessorSubType()
Like the previous entry, from Java we could call getconf, or look
under /proc/cpuinfo on linux. I assume there are other
o ImageProcess.getEnvironment()
Can be comfortable implemented using System.getenv()
o ImageAddressSpace.getSections()
All I can think of is the /proc/$$/maps file. Otherwise
To introspect the system more thoroughly JNI could be used to query the
OS directly:
o ImageProcess.getID()
A JNI function to call "getpid()" on UNIX or Linux platforms and
equivalent elsewhere.
The particularly painful points will be:
o ImageSection
In general, I don't see how these can be implemented without
knowledge from the JVM.
o ImageThread
Retrieving these and correlating them with JavaThread instances
is going to be difficult.
A Java thread executing a JNI thread could correlate a native
thread and Java thread, but
all of the other threads will have difficulty.
The difficulties above will all be shared by the CJVMTI implementation,
of course.
I won't go into more detail, but I think that introspecting from within
Java is not going to get us too far. Using JVMTI should produce
more useful information, and supplementing that with other functions to
introspect on the system itself would also be used by the CJVMTI
implementation of the RI. Of course, we might have two codebases that do
much the same thing to check against one another, so I think that
having explicit tests like we do just now (if only they were working...)
would still be prudent to eliminate the possibility of consistent errors and
ensure compliance with the specification.
Regards,
Stuart
Bobrovsky, Konstantin S wrote:
> Hi Stuart,
>
> one of the approaches to write TCK tests for the "1.0" mode I can think of
> is making the test application retrieve an log information to be checked
> (using standard Java API means) to some "golden" file, then expecting the
> TCK test obtain the same (or "similar") information from a "snapshot" using
> the RI API.
>
> In more details:
> There are several agents:
> - KATO implementation to be tested
> - Java Runtime which runs the tests and is coupled with the KATO
> implementation
> - test application (a set of classes run as a sample payload application)
> - TCK test, which can work in 2 modes:
> (1) run the test application, retrieve and log all the information
> to be checked in mode (2) to a "golden file" (should be stable
> between invokations), have the snapshot to be generated by some
> means.
> (2) use RI to read the snapshot and retrieve necessary information, then
> Compare what's retrieved with the golden file
> Each such test is run 2 times: in mode (1) to generate the golden file and
> the dump, and in mode (2) when actual API implementation verification
> happens.
>
> Of course, not entire API can be tested this way, but a fair part can. The main question is what kind of information can be obtained in mode (1), which can later be retrieved from the snapshot. What comes to mind is:
> - all alive threads running in the test application and their Java stacks
> using java.lang.Thread.getAllStackTraces. For threads with pre-defined
> names which the test recognizes as "special" ones (and which are
> created by the test application), some field values or other details
> can be queried
> - a subset of live objects created by the test application
> - ... many more if JVMTI or JMX is used as the information provider
> in mode (1)
>
> Thanks,
> Konst
>
> Intel Novosibirsk
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park,
> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
> Russian Federation
>
>
>
>> -----Original Message-----
>> From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>> Sent: Thursday, October 22, 2009 9:51 PM
>> To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>> Subject: TCK and all that
>>
>> Hi everybody,
>>
>> Apologies for the cross posting - but the implementation of the TCK
>> is relevant to both camps.
>> One of the things we need to do before making a release available is to
>> produce a TCK. By "TCK" I mean
>> a test harness that will:
>>
>> 1. Setup a JVM into a known configuration and generate a dump.
>> 2. Open a dump with the Kato API and run unit tests against it.
>>
>> Whether or not these are just functional tests or technology compliance
>> is something I will just gloss over for now
>> as just a classification issue.
>>
>> As we all know, there is a problem with optionality in the API. There is
>> information that will be lost when using
>> certain dump types.
>>
>> Steve has suggested two modes for running the TCK.
>>
>> Legacy
>> ========
>>
>> This mode runs through the API and confirms that the basic API behaves
>> as it is intended to do so.
>> It checks that the rules of the API are adhered to, but doesn't expect
>> any method will return any
>> particular piece of information.
>>
>> For example:
>> List<ImageProcess> ImageAddressSpace.getProcesses();
>>
>> processes would be checked to ensure that it is not null. If there were
>> contents
>> they would be queried, and their methods executed.
>>
>> For example:
>> String ImageProcess.getID();
>>
>> Would be expected to return a String or throw a DataUnavailable or
>> CorruptDataException exception.
>> The actual contents of the string wouldn't be checked - just so long as
>> it isn't null.
>> (incidentally, actually checking the value in this example for
>> correctness is difficult).
>>
>> 1.0
>> =====
>>
>> This is a test for exact compliance with the API. All of the information
>> that can be retrieved from the dump
>> must be retrieved and it must all be correct.
>>
>> With some caveats.
>>
>> Like with ImageProcess.getID(), there is some information that is poorly
>> specified or difficult to know in
>> advance.
>>
>> The poorly specified aspects of the API are:
>> Platform specific items.
>> For example:
>>
>> Implementation specific items:
>> toString() methods - what do they print?
>>
>> Things that are difficult to know in advance:
>> Native stack frames - ordering and naming.
>> JavaThreads - there will be threads that are implementation specific.
>>
>> There problems aren't intractable. A certain amount of flexibility will
>> be necessary to accommodate variations.
>> We should expect there to be more than what we expect, so we should just
>> look for what we expect to be there,
>> and ignore the rest.
>>
>> The poorly specified aspects of the API should just be more precisely
>> specified. Although it means that the TCK will
>> have to be aware of the different platforms and their behaviour.
>>
>>
>> I'm not a great fan of the "Legacy" mode. While it may have a basic use for
>> testing basic behaviour, I would like for there to be stronger statement
>> about what we expect the API to do.
>>
>> For example, if we call JavaRuntime.getThreads(), I would expect a
>> List<JavaThread> to be returned. However, if it wasn't empty, I would want
>> the threads we put into the testcase to be there and identifiable by their
>> (correct) names.
>>
>> i.e. if there is something to test we should test it.
>>
>> I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>> liberal, tolerant)
>>
>> There are, of course, problems. But I would expect that is this mode wasn't
>> possible or advisable, it would be improbable that a program could be
>> writtent to call the API.
>>
>> The end result is that if you get information out of the API, it must be
>> correct. Where you don't, it is ignored.
>>
>> Thoughts?
>>
>>
>> Thoughts?
>>
>> Regards,
>> Stuart
>>
>>
>>
>> --
>> Stuart Monteith
>> http://blog.stoo.me.uk/
>>
>
>
--
Stuart Monteith
http://blog.stoo.me.uk/
RE: TCK and all that
Posted by "Bobrovsky, Konstantin S" <ko...@intel.com>.
Hi Stuart,
one of the approaches to write TCK tests for the "1.0" mode I can think of
is making the test application retrieve an log information to be checked
(using standard Java API means) to some "golden" file, then expecting the
TCK test obtain the same (or "similar") information from a "snapshot" using
the RI API.
In more details:
There are several agents:
- KATO implementation to be tested
- Java Runtime which runs the tests and is coupled with the KATO
implementation
- test application (a set of classes run as a sample payload application)
- TCK test, which can work in 2 modes:
(1) run the test application, retrieve and log all the information
to be checked in mode (2) to a "golden file" (should be stable
between invokations), have the snapshot to be generated by some
means.
(2) use RI to read the snapshot and retrieve necessary information, then
Compare what's retrieved with the golden file
Each such test is run 2 times: in mode (1) to generate the golden file and
the dump, and in mode (2) when actual API implementation verification
happens.
Of course, not entire API can be tested this way, but a fair part can. The main question is what kind of information can be obtained in mode (1), which can later be retrieved from the snapshot. What comes to mind is:
- all alive threads running in the test application and their Java stacks
using java.lang.Thread.getAllStackTraces. For threads with pre-defined
names which the test recognizes as "special" ones (and which are
created by the test application), some field values or other details
can be queried
- a subset of live objects created by the test application
- ... many more if JVMTI or JMX is used as the information provider
in mode (1)
Thanks,
Konst
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
>-----Original Message-----
>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>Sent: Thursday, October 22, 2009 9:51 PM
>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>Subject: TCK and all that
>
>Hi everybody,
>
> Apologies for the cross posting - but the implementation of the TCK
>is relevant to both camps.
>One of the things we need to do before making a release available is to
>produce a TCK. By "TCK" I mean
>a test harness that will:
>
>1. Setup a JVM into a known configuration and generate a dump.
>2. Open a dump with the Kato API and run unit tests against it.
>
>Whether or not these are just functional tests or technology compliance
>is something I will just gloss over for now
>as just a classification issue.
>
>As we all know, there is a problem with optionality in the API. There is
>information that will be lost when using
>certain dump types.
>
>Steve has suggested two modes for running the TCK.
>
> Legacy
>========
>
>This mode runs through the API and confirms that the basic API behaves
>as it is intended to do so.
>It checks that the rules of the API are adhered to, but doesn't expect
>any method will return any
>particular piece of information.
>
>For example:
> List<ImageProcess> ImageAddressSpace.getProcesses();
>
>processes would be checked to ensure that it is not null. If there were
>contents
>they would be queried, and their methods executed.
>
>For example:
> String ImageProcess.getID();
>
>Would be expected to return a String or throw a DataUnavailable or
>CorruptDataException exception.
>The actual contents of the string wouldn't be checked - just so long as
>it isn't null.
>(incidentally, actually checking the value in this example for
>correctness is difficult).
>
> 1.0
>=====
>
>This is a test for exact compliance with the API. All of the information
>that can be retrieved from the dump
>must be retrieved and it must all be correct.
>
>With some caveats.
>
>Like with ImageProcess.getID(), there is some information that is poorly
>specified or difficult to know in
>advance.
>
>The poorly specified aspects of the API are:
> Platform specific items.
> For example:
>
> Implementation specific items:
> toString() methods - what do they print?
>
>Things that are difficult to know in advance:
> Native stack frames - ordering and naming.
> JavaThreads - there will be threads that are implementation specific.
>
>There problems aren't intractable. A certain amount of flexibility will
>be necessary to accommodate variations.
>We should expect there to be more than what we expect, so we should just
>look for what we expect to be there,
>and ignore the rest.
>
>The poorly specified aspects of the API should just be more precisely
>specified. Although it means that the TCK will
>have to be aware of the different platforms and their behaviour.
>
>
>I'm not a great fan of the "Legacy" mode. While it may have a basic use for
>testing basic behaviour, I would like for there to be stronger statement
>about what we expect the API to do.
>
>For example, if we call JavaRuntime.getThreads(), I would expect a
>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>the threads we put into the testcase to be there and identifiable by their
>(correct) names.
>
>i.e. if there is something to test we should test it.
>
>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>liberal, tolerant)
>
>There are, of course, problems. But I would expect that is this mode wasn't
>possible or advisable, it would be improbable that a program could be
>writtent to call the API.
>
>The end result is that if you get information out of the API, it must be
>correct. Where you don't, it is ignored.
>
>Thoughts?
>
>
>Thoughts?
>
>Regards,
> Stuart
>
>
>
>--
>Stuart Monteith
>http://blog.stoo.me.uk/
RE: TCK and all that
Posted by "Bobrovsky, Konstantin S" <ko...@intel.com>.
Hi Stuart,
one of the approaches to write TCK tests for the "1.0" mode I can think of
is making the test application retrieve an log information to be checked
(using standard Java API means) to some "golden" file, then expecting the
TCK test obtain the same (or "similar") information from a "snapshot" using
the RI API.
In more details:
There are several agents:
- KATO implementation to be tested
- Java Runtime which runs the tests and is coupled with the KATO
implementation
- test application (a set of classes run as a sample payload application)
- TCK test, which can work in 2 modes:
(1) run the test application, retrieve and log all the information
to be checked in mode (2) to a "golden file" (should be stable
between invokations), have the snapshot to be generated by some
means.
(2) use RI to read the snapshot and retrieve necessary information, then
Compare what's retrieved with the golden file
Each such test is run 2 times: in mode (1) to generate the golden file and
the dump, and in mode (2) when actual API implementation verification
happens.
Of course, not entire API can be tested this way, but a fair part can. The main question is what kind of information can be obtained in mode (1), which can later be retrieved from the snapshot. What comes to mind is:
- all alive threads running in the test application and their Java stacks
using java.lang.Thread.getAllStackTraces. For threads with pre-defined
names which the test recognizes as "special" ones (and which are
created by the test application), some field values or other details
can be queried
- a subset of live objects created by the test application
- ... many more if JVMTI or JMX is used as the information provider
in mode (1)
Thanks,
Konst
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
>-----Original Message-----
>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>Sent: Thursday, October 22, 2009 9:51 PM
>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>Subject: TCK and all that
>
>Hi everybody,
>
> Apologies for the cross posting - but the implementation of the TCK
>is relevant to both camps.
>One of the things we need to do before making a release available is to
>produce a TCK. By "TCK" I mean
>a test harness that will:
>
>1. Setup a JVM into a known configuration and generate a dump.
>2. Open a dump with the Kato API and run unit tests against it.
>
>Whether or not these are just functional tests or technology compliance
>is something I will just gloss over for now
>as just a classification issue.
>
>As we all know, there is a problem with optionality in the API. There is
>information that will be lost when using
>certain dump types.
>
>Steve has suggested two modes for running the TCK.
>
> Legacy
>========
>
>This mode runs through the API and confirms that the basic API behaves
>as it is intended to do so.
>It checks that the rules of the API are adhered to, but doesn't expect
>any method will return any
>particular piece of information.
>
>For example:
> List<ImageProcess> ImageAddressSpace.getProcesses();
>
>processes would be checked to ensure that it is not null. If there were
>contents
>they would be queried, and their methods executed.
>
>For example:
> String ImageProcess.getID();
>
>Would be expected to return a String or throw a DataUnavailable or
>CorruptDataException exception.
>The actual contents of the string wouldn't be checked - just so long as
>it isn't null.
>(incidentally, actually checking the value in this example for
>correctness is difficult).
>
> 1.0
>=====
>
>This is a test for exact compliance with the API. All of the information
>that can be retrieved from the dump
>must be retrieved and it must all be correct.
>
>With some caveats.
>
>Like with ImageProcess.getID(), there is some information that is poorly
>specified or difficult to know in
>advance.
>
>The poorly specified aspects of the API are:
> Platform specific items.
> For example:
>
> Implementation specific items:
> toString() methods - what do they print?
>
>Things that are difficult to know in advance:
> Native stack frames - ordering and naming.
> JavaThreads - there will be threads that are implementation specific.
>
>There problems aren't intractable. A certain amount of flexibility will
>be necessary to accommodate variations.
>We should expect there to be more than what we expect, so we should just
>look for what we expect to be there,
>and ignore the rest.
>
>The poorly specified aspects of the API should just be more precisely
>specified. Although it means that the TCK will
>have to be aware of the different platforms and their behaviour.
>
>
>I'm not a great fan of the "Legacy" mode. While it may have a basic use for
>testing basic behaviour, I would like for there to be stronger statement
>about what we expect the API to do.
>
>For example, if we call JavaRuntime.getThreads(), I would expect a
>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>the threads we put into the testcase to be there and identifiable by their
>(correct) names.
>
>i.e. if there is something to test we should test it.
>
>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>liberal, tolerant)
>
>There are, of course, problems. But I would expect that is this mode wasn't
>possible or advisable, it would be improbable that a program could be
>writtent to call the API.
>
>The end result is that if you get information out of the API, it must be
>correct. Where you don't, it is ignored.
>
>Thoughts?
>
>
>Thoughts?
>
>Regards,
> Stuart
>
>
>
>--
>Stuart Monteith
>http://blog.stoo.me.uk/