You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Weldon Washburn <we...@gmail.com> on 2007/01/01 02:11:54 UTC
Re: [testing] Reliability test suite is contributed, HARMONY-2918
On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
>
> Hi,
>
> Reliability test suite is contributed on behalf of Intel:
>
> http://issues.apache.org/jira/browse/HARMONY-2918
>
> The tests are intended to verify ability of the runtime to run
> successfully for long time. The suite contains 72 tests (covering
> mainly kernel, nio, text, io, net, zip classes functionality, plus a
> couple of tests for VM) and ant scripts to build and run tests.
>
> The tests are intented to be integrated into buildtest/. Please review
> and comment.
This is great. These are much needed tests.
I have started looking at the tests and have questions about
kernel/thread/VolatileVariableTest/DekkerTest.java.
regionIn() corresponds to the part of the Dekker algorithm that acquires the
critical section. regionIn() contains code that, as far as I can tell, is
not part of the Dekker algorithm. regionIn() spins for 100000 loops then
assumes/pretends the critical section has been acquired (??). The line of
code I refer to is:
while (current != worker && k < 100000) {k++;}
>From reading the Dekker algo reference, it seems this line of code probably
should be:
while (current != worker) { }
The burning question is why was the "k < 100000" test added?
At the top of DekkerTest.java it says, "Goal: check that VM supports
volatile variables". My guess is that the JIT must put memory fence(s)
around volatile variable reads and writes. My guess is that Dekker is
probably one of the few ways to actually test for proper implementation.
But I really worry that the JIT could accidentally miss emitting the memory
fence(s). And DekkerTest.java would pass anyway. Does it make sense to
create "DekkerTest2.java" that is identical to DekkerTest.java where all the
"volatile" kepwords are omitted? The intention is to create a test that
will hard fail if the memory fences are not emitted.
Best regards, Oleg
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by tatyana doubtsova <ta...@gmail.com>.
Weldon,
I've looked at suggested DekkerTest.
I think that k < 100000 should be eliminated. Thanks for pointing it out.
Also I think that test instability root cause is volatile array usage, which
should be replaced by volatile variables.
I've attached another version of DekkerTest.java to
http://issues.apache.org/jira/browse/HARMONY-2918
This test intermittently hangs on drlvm, see
https://issues.apache.org/jira/browse/HARMONY-2986
for detailes.
Thanks,
Tanya
On 1/2/07, Weldon Washburn <we...@gmail.com> wrote:
>
> On 1/2/07, Oleg Oleinik <ol...@gmail.com> wrote:
> >
> > > while (current != worker && k < 100000) {k++;}
> > > The burning question is why was the "k < 100000" test added?
> >
> > Look like "k<100000" is added to avoid test hanging, looks like it
> > should be removed. Let me think more.
>
>
> I also suspect the "k<1000000" was added to avoid test hanging. If the
> test does, in fact, hang when "k<1000000" is removed, we have a whole new
> set of problems to deal with. It could be the case that this test
> "reliably" passes with a known race condition. In other words, this test
> may indicate bugs in the test as well as the JVM. My experience is that
> these kinds of issues can be very difficult and time consuming as they
> involve the hardware's memory consistency model as well as the software
> running on top.
>
> On 1/1/07, Weldon Washburn <we...@gmail.com> wrote:
> > > On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Reliability test suite is contributed on behalf of Intel:
> > > >
> > > > http://issues.apache.org/jira/browse/HARMONY-2918
> > > >
> > > > The tests are intended to verify ability of the runtime to run
> > > > successfully for long time. The suite contains 72 tests (covering
> > > > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > > > couple of tests for VM) and ant scripts to build and run tests.
> > > >
> > > > The tests are intented to be integrated into buildtest/. Please
> review
> > > > and comment.
> > >
> > >
> > > This is great. These are much needed tests.
> > >
> > > I have started looking at the tests and have questions about
> > > kernel/thread/VolatileVariableTest/DekkerTest.java.
> > >
> > > regionIn() corresponds to the part of the Dekker algorithm that
> acquires
> > the
> > > critical section. regionIn() contains code that, as far as I can
> tell,
> > is
> > > not part of the Dekker algorithm. regionIn() spins for 100000 loops
> > then
> > > assumes/pretends the critical section has been acquired (??). The
> line
> > of
> > > code I refer to is:
> > >
> > > while (current != worker && k < 100000) {k++;}
> > >
> > > From reading the Dekker algo reference, it seems this line of code
> > probably
> > > should be:
> > >
> > > while (current != worker) { }
> > >
> > > The burning question is why was the "k < 100000" test added?
> > >
> > > At the top of DekkerTest.java it says, "Goal: check that VM supports
> > > volatile variables". My guess is that the JIT must put memory
> fence(s)
> > > around volatile variable reads and writes. My guess is that Dekker is
> > > probably one of the few ways to actually test for proper
> implementation.
> > > But I really worry that the JIT could accidentally miss emitting the
> > memory
> > > fence(s). And DekkerTest.java would pass anyway. Does it make sense
> to
> > > create "DekkerTest2.java" that is identical to DekkerTest.java where
> all
> > the
> > > "volatile" kepwords are omitted? The intention is to create a test
> that
> > > will hard fail if the memory fences are not emitted.
> > >
> > >
> > > Best regards, Oleg
> > > >
> > >
> > >
> > >
> > > --
> > > Weldon Washburn
> > > Intel Enterprise Solutions Software Division
> > >
> > >
> >
>
>
>
> --
> Weldon Washburn
> Intel Enterprise Solutions Software Division
>
>
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by Weldon Washburn <we...@gmail.com>.
On 1/2/07, Oleg Oleinik <ol...@gmail.com> wrote:
>
> > while (current != worker && k < 100000) {k++;}
> > The burning question is why was the "k < 100000" test added?
>
> Look like "k<100000" is added to avoid test hanging, looks like it
> should be removed. Let me think more.
I also suspect the "k<1000000" was added to avoid test hanging. If the
test does, in fact, hang when "k<1000000" is removed, we have a whole new
set of problems to deal with. It could be the case that this test
"reliably" passes with a known race condition. In other words, this test
may indicate bugs in the test as well as the JVM. My experience is that
these kinds of issues can be very difficult and time consuming as they
involve the hardware's memory consistency model as well as the software
running on top.
On 1/1/07, Weldon Washburn <we...@gmail.com> wrote:
> > On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > Reliability test suite is contributed on behalf of Intel:
> > >
> > > http://issues.apache.org/jira/browse/HARMONY-2918
> > >
> > > The tests are intended to verify ability of the runtime to run
> > > successfully for long time. The suite contains 72 tests (covering
> > > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > > couple of tests for VM) and ant scripts to build and run tests.
> > >
> > > The tests are intented to be integrated into buildtest/. Please review
> > > and comment.
> >
> >
> > This is great. These are much needed tests.
> >
> > I have started looking at the tests and have questions about
> > kernel/thread/VolatileVariableTest/DekkerTest.java.
> >
> > regionIn() corresponds to the part of the Dekker algorithm that acquires
> the
> > critical section. regionIn() contains code that, as far as I can tell,
> is
> > not part of the Dekker algorithm. regionIn() spins for 100000 loops
> then
> > assumes/pretends the critical section has been acquired (??). The line
> of
> > code I refer to is:
> >
> > while (current != worker && k < 100000) {k++;}
> >
> > From reading the Dekker algo reference, it seems this line of code
> probably
> > should be:
> >
> > while (current != worker) { }
> >
> > The burning question is why was the "k < 100000" test added?
> >
> > At the top of DekkerTest.java it says, "Goal: check that VM supports
> > volatile variables". My guess is that the JIT must put memory fence(s)
> > around volatile variable reads and writes. My guess is that Dekker is
> > probably one of the few ways to actually test for proper implementation.
> > But I really worry that the JIT could accidentally miss emitting the
> memory
> > fence(s). And DekkerTest.java would pass anyway. Does it make sense to
> > create "DekkerTest2.java" that is identical to DekkerTest.java where all
> the
> > "volatile" kepwords are omitted? The intention is to create a test that
> > will hard fail if the memory fences are not emitted.
> >
> >
> > Best regards, Oleg
> > >
> >
> >
> >
> > --
> > Weldon Washburn
> > Intel Enterprise Solutions Software Division
> >
> >
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by Oleg Oleinik <ol...@gmail.com>.
> while (current != worker && k < 100000) {k++;}
> The burning question is why was the "k < 100000" test added?
Look like "k<100000" is added to avoid test hanging, looks like it
should be removed. Let me think more.
On 1/1/07, Weldon Washburn <we...@gmail.com> wrote:
> On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> >
> > Hi,
> >
> > Reliability test suite is contributed on behalf of Intel:
> >
> > http://issues.apache.org/jira/browse/HARMONY-2918
> >
> > The tests are intended to verify ability of the runtime to run
> > successfully for long time. The suite contains 72 tests (covering
> > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > couple of tests for VM) and ant scripts to build and run tests.
> >
> > The tests are intented to be integrated into buildtest/. Please review
> > and comment.
>
>
> This is great. These are much needed tests.
>
> I have started looking at the tests and have questions about
> kernel/thread/VolatileVariableTest/DekkerTest.java.
>
> regionIn() corresponds to the part of the Dekker algorithm that acquires the
> critical section. regionIn() contains code that, as far as I can tell, is
> not part of the Dekker algorithm. regionIn() spins for 100000 loops then
> assumes/pretends the critical section has been acquired (??). The line of
> code I refer to is:
>
> while (current != worker && k < 100000) {k++;}
>
> From reading the Dekker algo reference, it seems this line of code probably
> should be:
>
> while (current != worker) { }
>
> The burning question is why was the "k < 100000" test added?
>
> At the top of DekkerTest.java it says, "Goal: check that VM supports
> volatile variables". My guess is that the JIT must put memory fence(s)
> around volatile variable reads and writes. My guess is that Dekker is
> probably one of the few ways to actually test for proper implementation.
> But I really worry that the JIT could accidentally miss emitting the memory
> fence(s). And DekkerTest.java would pass anyway. Does it make sense to
> create "DekkerTest2.java" that is identical to DekkerTest.java where all the
> "volatile" kepwords are omitted? The intention is to create a test that
> will hard fail if the memory fences are not emitted.
>
>
> Best regards, Oleg
> >
>
>
>
> --
> Weldon Washburn
> Intel Enterprise Solutions Software Division
>
>
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x252 day of Apache Harmony Weldon Washburn wrote:
> On 02 Jan 2007 18:17:37 +0600, Egor Pasko <eg...@gmail.com> wrote:
> >
> > On the 0x251 day of Apache Harmony Weldon Washburn wrote:
> > > On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Reliability test suite is contributed on behalf of Intel:
> > > >
> > > > http://issues.apache.org/jira/browse/HARMONY-2918
> > > >
> > > > The tests are intended to verify ability of the runtime to run
> > > > successfully for long time. The suite contains 72 tests (covering
> > > > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > > > couple of tests for VM) and ant scripts to build and run tests.
> > > >
> > > > The tests are intented to be integrated into buildtest/. Please review
> > > > and comment.
> > >
> > >
> > > This is great. These are much needed tests.
> > >
> > > I have started looking at the tests and have questions about
> > > kernel/thread/VolatileVariableTest/DekkerTest.java.
> > >
> > > regionIn() corresponds to the part of the Dekker algorithm that acquires
> > the
> > > critical section. regionIn() contains code that, as far as I can tell,
> > is
> > > not part of the Dekker algorithm. regionIn() spins for 100000 loops
> > then
> > > assumes/pretends the critical section has been acquired (??). The line
> > of
> > > code I refer to is:
> > >
> > > while (current != worker && k < 100000) {k++;}
> > >
> > > From reading the Dekker algo reference, it seems this line of code
> > probably
> > > should be:
> > >
> > > while (current != worker) { }
> > >
> > > The burning question is why was the "k < 100000" test added?
> > >
> > > At the top of DekkerTest.java it says, "Goal: check that VM supports
> > > volatile variables". My guess is that the JIT must put memory fence(s)
> > > around volatile variable reads and writes. My guess is that Dekker is
> > > probably one of the few ways to actually test for proper implementation.
> > > But I really worry that the JIT could accidentally miss emitting the
> > memory
> > > fence(s).
> >
> > why? In JIT we cancel "Common Subexpression Elimination"-like
> > optimizations for volatile variables, which sould be enough for now.
>
>
> Eliminating CSE optimization is good. I have not thought about it too
> closely but the JIT may actually need to emit read and/or write fences also.
>
> Another issue is preventing regression over the long term. Someone might
> accidentally and unintentionally turn on CSE for volatile variables. While
> it may be too idealistic, it would be real nice to have a test suite pick
> this up.
>
>
> > And DekkerTest.java would pass anyway. Does it make sense to create
> > > "DekkerTest2.java" that is identical to DekkerTest.java where all
> > > the "volatile" kepwords are omitted? The intention is to create a
> > > test that will hard fail if the memory fences are not emitted.
> >
> > Weldon, interesting idea! But DekkerTest2.java will not necessarily
> > fail, especially for JET because JET almost does NOT optimize memory
> > accesses.
>
>
> Good point. But JET might also need to emit memory fence(s).
IIRC, mem fences are necessary only on monenter/monexit. JET does not
reuse registers accross calls. monenter/monexit is a call for JET. So,
memory fences are at the place with this scheme.
> And the existing test may not be strong enough to show that fences
> are missing. Then again maybe the existing test is, in fact,
> showing us the fences are missing. In specific, maybe the
> "k<1000000" issue would evaporate if memory fences were inserted.
testing mem fences and volatile variables coherency is something
useful. I hope to see some more tests of the kind with the TCK.
> That makes me think DekkerTest2.java would show us how
> > DekkerTest is good at testing OPT. But... DekkerTest2 would NOT be
> > suitable for regular runs.
>
>
> Good point. DekkerTest2 is a half-baked idea intended to get conversation
> started. What's really needed is a test that specifically verifies each and
> every volatile read/write goes all the way into the hardware coherency
> domain on an SMP. I am trying to think of a simple way of doing this.
> Maybe the following half-baked idea can be made to work:
>
> static volatile int v1 = 0;
> static boolean b1, b2 = false;
>
> //// Thread 0 runs the following loop endlessly, the coherency domain
> should see v1 "flicker"
> while(true) {
> v1 = 1;
> v1 = 2; // with write fence removed and CSE turned on, RAM version
> of v1 == 0 (I think)
> }
> -----------------------------------
> //// Thread 1 verifies memory "flicker"
> for (int ss = 0; ss < 1000 * 1000 * 10; ss++) {
> if (v1 == 1) b1 = true;
> if (v1 == 2) b2 = true;
> }
> if ( (b1 == true) && (b2 == true) ) return TEST.PASSED;
> return TEST.FAILED;
>
> I suspect any volatile variable test will always pass OK on a UP. These
> kinds of tests need to be run on both 2-way and 4-way. There are classes of
> multiprocessor bugs that are impossible to observe on a 2-way and impossible
> to observe using only 2 threads. In other words, we probably have to go to
> 4+ threads on a 4-way to really nail down the regression tests.
Weldon, what are we searching for? hardware bugs or JIT bugs?
JIT bug in incorrect fencing of 'v' can be easily pinpointed by your
test with ANY CORRECT SMP hardware.
--
Egor Pasko
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by Weldon Washburn <we...@gmail.com>.
On 02 Jan 2007 18:17:37 +0600, Egor Pasko <eg...@gmail.com> wrote:
>
> On the 0x251 day of Apache Harmony Weldon Washburn wrote:
> > On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > Reliability test suite is contributed on behalf of Intel:
> > >
> > > http://issues.apache.org/jira/browse/HARMONY-2918
> > >
> > > The tests are intended to verify ability of the runtime to run
> > > successfully for long time. The suite contains 72 tests (covering
> > > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > > couple of tests for VM) and ant scripts to build and run tests.
> > >
> > > The tests are intented to be integrated into buildtest/. Please review
> > > and comment.
> >
> >
> > This is great. These are much needed tests.
> >
> > I have started looking at the tests and have questions about
> > kernel/thread/VolatileVariableTest/DekkerTest.java.
> >
> > regionIn() corresponds to the part of the Dekker algorithm that acquires
> the
> > critical section. regionIn() contains code that, as far as I can tell,
> is
> > not part of the Dekker algorithm. regionIn() spins for 100000 loops
> then
> > assumes/pretends the critical section has been acquired (??). The line
> of
> > code I refer to is:
> >
> > while (current != worker && k < 100000) {k++;}
> >
> > From reading the Dekker algo reference, it seems this line of code
> probably
> > should be:
> >
> > while (current != worker) { }
> >
> > The burning question is why was the "k < 100000" test added?
> >
> > At the top of DekkerTest.java it says, "Goal: check that VM supports
> > volatile variables". My guess is that the JIT must put memory fence(s)
> > around volatile variable reads and writes. My guess is that Dekker is
> > probably one of the few ways to actually test for proper implementation.
> > But I really worry that the JIT could accidentally miss emitting the
> memory
> > fence(s).
>
> why? In JIT we cancel "Common Subexpression Elimination"-like
> optimizations for volatile variables, which sould be enough for now.
Eliminating CSE optimization is good. I have not thought about it too
closely but the JIT may actually need to emit read and/or write fences also.
Another issue is preventing regression over the long term. Someone might
accidentally and unintentionally turn on CSE for volatile variables. While
it may be too idealistic, it would be real nice to have a test suite pick
this up.
> And DekkerTest.java would pass anyway. Does it make sense to create
> > "DekkerTest2.java" that is identical to DekkerTest.java where all
> > the "volatile" kepwords are omitted? The intention is to create a
> > test that will hard fail if the memory fences are not emitted.
>
> Weldon, interesting idea! But DekkerTest2.java will not necessarily
> fail, especially for JET because JET almost does NOT optimize memory
> accesses.
Good point. But JET might also need to emit memory fence(s). And the
existing test may not be strong enough to show that fences are missing.
Then again maybe the existing test is, in fact, showing us the fences are
missing. In specific, maybe the "k<1000000" issue would evaporate if memory
fences were inserted.
That makes me think DekkerTest2.java would show us how
> DekkerTest is good at testing OPT. But... DekkerTest2 would NOT be
> suitable for regular runs.
Good point. DekkerTest2 is a half-baked idea intended to get conversation
started. What's really needed is a test that specifically verifies each and
every volatile read/write goes all the way into the hardware coherency
domain on an SMP. I am trying to think of a simple way of doing this.
Maybe the following half-baked idea can be made to work:
static volatile int v1 = 0;
static boolean b1, b2 = false;
//// Thread 0 runs the following loop endlessly, the coherency domain
should see v1 "flicker"
while(true) {
v1 = 1;
v1 = 2; // with write fence removed and CSE turned on, RAM version
of v1 == 0 (I think)
}
-----------------------------------
//// Thread 1 verifies memory "flicker"
for (int ss = 0; ss < 1000 * 1000 * 10; ss++) {
if (v1 == 1) b1 = true;
if (v1 == 2) b2 = true;
}
if ( (b1 == true) && (b2 == true) ) return TEST.PASSED;
return TEST.FAILED;
I suspect any volatile variable test will always pass OK on a UP. These
kinds of tests need to be run on both 2-way and 4-way. There are classes of
multiprocessor bugs that are impossible to observe on a 2-way and impossible
to observe using only 2 threads. In other words, we probably have to go to
4+ threads on a 4-way to really nail down the regression tests.
Alternative idea to check 'volatile' goodness in OPT: comment out all
> volatile checks in Jitrino.OPT's sources. You will see how real
> multithreaded apps like Eclipse fail to work without special
> 'volatile' handling (we call it memory fences).
>
> --
> Egor Pasko
>
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [testing] Reliability test suite is contributed, HARMONY-2918
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x251 day of Apache Harmony Weldon Washburn wrote:
> On 12/28/06, Oleg Oleinik <ol...@gmail.com> wrote:
> >
> > Hi,
> >
> > Reliability test suite is contributed on behalf of Intel:
> >
> > http://issues.apache.org/jira/browse/HARMONY-2918
> >
> > The tests are intended to verify ability of the runtime to run
> > successfully for long time. The suite contains 72 tests (covering
> > mainly kernel, nio, text, io, net, zip classes functionality, plus a
> > couple of tests for VM) and ant scripts to build and run tests.
> >
> > The tests are intented to be integrated into buildtest/. Please review
> > and comment.
>
>
> This is great. These are much needed tests.
>
> I have started looking at the tests and have questions about
> kernel/thread/VolatileVariableTest/DekkerTest.java.
>
> regionIn() corresponds to the part of the Dekker algorithm that acquires the
> critical section. regionIn() contains code that, as far as I can tell, is
> not part of the Dekker algorithm. regionIn() spins for 100000 loops then
> assumes/pretends the critical section has been acquired (??). The line of
> code I refer to is:
>
> while (current != worker && k < 100000) {k++;}
>
> From reading the Dekker algo reference, it seems this line of code probably
> should be:
>
> while (current != worker) { }
>
> The burning question is why was the "k < 100000" test added?
>
> At the top of DekkerTest.java it says, "Goal: check that VM supports
> volatile variables". My guess is that the JIT must put memory fence(s)
> around volatile variable reads and writes. My guess is that Dekker is
> probably one of the few ways to actually test for proper implementation.
> But I really worry that the JIT could accidentally miss emitting the memory
> fence(s).
why? In JIT we cancel "Common Subexpression Elimination"-like
optimizations for volatile variables, which sould be enough for now.
> And DekkerTest.java would pass anyway. Does it make sense to create
> "DekkerTest2.java" that is identical to DekkerTest.java where all
> the "volatile" kepwords are omitted? The intention is to create a
> test that will hard fail if the memory fences are not emitted.
Weldon, interesting idea! But DekkerTest2.java will not necessarily
fail, especially for JET because JET almost does NOT optimize memory
accesses. That makes me think DekkerTest2.java would show us how
DekkerTest is good at testing OPT. But... DekkerTest2 would NOT be
suitable for regular runs.
Alternative idea to check 'volatile' goodness in OPT: comment out all
volatile checks in Jitrino.OPT's sources. You will see how real
multithreaded apps like Eclipse fail to work without special
'volatile' handling (we call it memory fences).
--
Egor Pasko