You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Bryan Pendleton <bp...@amberpoint.com> on 2007/04/13 00:31:01 UTC
Question about UpdateCursorTest.testVirtualMemoryHeap
I'm a bit confused about the test program
UpdateCursorTest.testVirtualMemoryHeap. This test program
appears to depend on the order of the rows returned, but
it does not contain an ORDER BY clause in the query. Here's
the code I'm looking at, from UpdateCursorTest.java:
/**
* Test the virtual memory heap.
*
* @throws SQLException
*/
public void testVirtualMemoryHeap() throws SQLException {
PreparedStatement select = prepareStatement("select c1, c3 from t1 where c3 > 1 and c1 > 0
for update");
Statement update = createStatement();
String cursorName;
ResultSet cursor;
int expectedValue = 1;
cursor = select.executeQuery(); // cursor is now open
cursorName = cursor.getCursorName();
/* scan the entire table except the last row. */
for (int i = 0; i < SIZE_OF_T1 - 1; i++) {
/* Notice the order in the rows we get: from 2 to 102 asc order on second column (c3)
* then from 202 down to 103 on that column; then from 203 up to 250. The reason is
* we are using asc index on c3, all the rows updated are in the future direction of the
* index scan, so they all get filled into a hash table. The MAX_MEMORY_PER_TABLE
* property determines max cap of hash table 100. So from row 103 it goes into virtual
* memory heap, whose in memory part is also 100 entries. So row 103 to 202 goes into
* the in-memory part and gets dumped out in reverse order. Finally Row 203 to 250"
* goes into file system. Here we mean row ids.
*/
This test seems to be very sensitive to the precise query execution
strategy that is being used, but I don't see how the test is
controlling that query execution strategy.
Can somebody clarify how the test works for me?
thanks,
bryan
P.S. The reason that I'm asking is that my proposed change for
DERBY-2526 causes this test to fail, but I'm not sure whether that
means my change is bad, or whether this test is depending on a
query execution strategy that is not guaranteed to occur.
Re: Question about UpdateCursorTest.testVirtualMemoryHeap
Posted by Bryan Pendleton <bp...@amberpoint.com>.
> As Mike pointed out, this test has been failing in the nightlies, so
> it might not be your change that is causing the failure.
Thanks Andrew and Mike! That was very helpful information.
bryan
Re: Question about UpdateCursorTest.testVirtualMemoryHeap
Posted by Andrew McIntyre <mc...@gmail.com>.
On 4/12/07, Bryan Pendleton <bp...@amberpoint.com> wrote:
>
> This test seems to be very sensitive to the precise query execution
> strategy that is being used, but I don't see how the test is
> controlling that query execution strategy.
>
> Can somebody clarify how the test works for me?
Perhaps this comment from the original (pre-JUnit) test should have
been kept as well:
Not done in ij since we need to do many "next" and "update" to be
able to excercise the code of creating temp conglomerate for virtual
memory heap. We need at minimum
200 rows in table, if "maxMemoryPerTable" property is set to 1 (KB).
This includes 100 rows to fill the hash table and another 100 rows
to fill the in-memory heap.
So the size of the rows for t1 cause the hashtable to fill after 100
rows, the in-memory heap after another 100, and then finally a few
more rows to go to the backing store. It appears to depend on the
behavior of the in-memory heap of reversing the order of the rows that
are put into it. I'm not entirely sure what the 'in-memory heap' here
is referring to, presumably this is the in-memory portion of a
BackingStoreHashtable?
As Mike pointed out, this test has been failing in the nightlies, so
it might not be your change that is causing the failure. Also
unfortunate is the fact that I can't reproduce this on my machine,
although I am working on that. The check for row order was added
because from the comments in the previous test, it was clear that the
row order was what was intended to be checked.
andrew
Re: Question about UpdateCursorTest.testVirtualMemoryHeap
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Mike Matrigali wrote:
>
>> I'm a bit confused about the test program
>> UpdateCursorTest.testVirtualMemoryHeap. This test program
>> appears to depend on the order of the rows returned, but
>> it does not contain an ORDER BY clause in the query. Here's
>> the code I'm looking at, from UpdateCursorTest.java:
>>
>> /**
>> * Test the virtual memory heap.
>> *
>> * @throws SQLException
>> */
>> public void testVirtualMemoryHeap() throws SQLException {
>> PreparedStatement select = prepareStatement("select c1, c3
>> from t1 where c3 > 1 and c1 > 0 for update");
>> Statement update = createStatement();
>> String cursorName;
>> ResultSet cursor;
>> int expectedValue = 1;
>>
>> cursor = select.executeQuery(); // cursor is now open
>> cursorName = cursor.getCursorName();
>>
>> /* scan the entire table except the last row. */
>> for (int i = 0; i < SIZE_OF_T1 - 1; i++) {
>>
>> /* Notice the order in the rows we get: from 2 to 102 asc
>> order on second column (c3)
>> * then from 202 down to 103 on that column; then from
>> 203 up to 250. The reason is
>> * we are using asc index on c3, all the rows updated are
>> in the future direction of the
>> * index scan, so they all get filled into a hash table.
>> The MAX_MEMORY_PER_TABLE
>> * property determines max cap of hash table 100. So
>> from row 103 it goes into virtual
>> * memory heap, whose in memory part is also 100
>> entries. So row 103 to 202 goes into
>> * the in-memory part and gets dumped out in reverse
>> order. Finally Row 203 to 250"
>> * goes into file system. Here we mean row ids.
>> */
I have not looked at test code or derby code, but the comments do seem
worrysome. Picking a specific plan can be dependent on machine speed,
amount of memory at the particular time the test is run (so very easily
could change as more tests are added to suite). Also the comments talk
about expecting a particular order when scanning a hash table - I know
that I have seen different order of rows coming out of java hash tables
depending on what JVM you are using.
I have filed DERBY-2543 to track the new regression suite bug.
>>
>> This test seems to be very sensitive to the precise query execution
>> strategy that is being used, but I don't see how the test is
>> controlling that query execution strategy.
>>
>> Can somebody clarify how the test works for me?
>>
>> thanks,
>>
>> bryan
>>
>> P.S. The reason that I'm asking is that my proposed change for
>> DERBY-2526 causes this test to fail, but I'm not sure whether that
>> means my change is bad, or whether this test is depending on a
>> query execution strategy that is not guaranteed to occur.
>>
>>
>>
>
>
>
Re: Question about UpdateCursorTest.testVirtualMemoryHeap
Posted by Mike Matrigali <mi...@sbcglobal.net>.
It looks like this test is failing in the current trunk without
your changes:
http://dbtg.thresher.com/derby/test/tinderbox_trunk16/jvm1.6/testing/testlog/SunOS-5.10_i86pc-i386/528002-org.apache.derbyTesting.functionTests.suites.All_diff.txt
a quick scan of the tinderbox reports here:
http://dbtg.thresher.com/derby/test/tinderbox_trunk16/jvm1.6/testing/Limited/index.html
And it looks like the test has been failing ever since it was checked in:
http://dbtg.thresher.com/derby/test/tinderbox_trunk16/jvm1.6/testing/testlog/SunOS-5.10_i86pc-i386/526070-org.apache.derbyTesting.functionTests.suites.All_diff.txt
Changes from 525953/526069 to 526070:
------------------------------------------------------------------------
r526070 | fuzzylogic | 2007-04-06 08:20:46 +0200 (Fri, 06 Apr 2007) | 4
lines
DERBY-2467: Convert lang/updateCursor.java to JUnit.
Committed for Ugo Matrangolo <ma...@santippe.dyndns.org>
------------------------------------------------------------------------
D
java/testing/org/apache/derbyTesting/functionTests/tests/lang/updateCursor.java
D
java/testing/org/apache/derbyTesting/functionTests/tests/lang/updateCursor_derby.properties
U
java/testing/org/apache/derbyTesting/functionTests/tests/lang/_Suite.java
A
java/testing/org/apache/derbyTesting/functionTests/tests/lang/UpdateCursorTest.java
D
java/testing/org/apache/derbyTesting/functionTests/master/updateCursor.out
U
java/testing/org/apache/derbyTesting/functionTests/suites/derbylang.runall
Fetching external item into 'tools/testing/derby/10.1'
Updated external to revision 526070.
Updated to revision 526070.
526070 URL: https://svn.apache.org/repos/asf/db/derby/code/trunk
Performing status on external item at 'tools/testing/derby/10.1'
From some comments in the checkin it looks like order test was added as
a requirement for checkin, but don't know if it is the specific one you
are highlighting.
Bryan Pendleton wrote:
> I'm a bit confused about the test program
> UpdateCursorTest.testVirtualMemoryHeap. This test program
> appears to depend on the order of the rows returned, but
> it does not contain an ORDER BY clause in the query. Here's
> the code I'm looking at, from UpdateCursorTest.java:
>
> /**
> * Test the virtual memory heap.
> *
> * @throws SQLException
> */
> public void testVirtualMemoryHeap() throws SQLException {
> PreparedStatement select = prepareStatement("select c1, c3 from
> t1 where c3 > 1 and c1 > 0 for update");
> Statement update = createStatement();
> String cursorName;
> ResultSet cursor;
> int expectedValue = 1;
>
> cursor = select.executeQuery(); // cursor is now open
> cursorName = cursor.getCursorName();
>
> /* scan the entire table except the last row. */
> for (int i = 0; i < SIZE_OF_T1 - 1; i++) {
>
> /* Notice the order in the rows we get: from 2 to 102 asc
> order on second column (c3)
> * then from 202 down to 103 on that column; then from 203
> up to 250. The reason is
> * we are using asc index on c3, all the rows updated are
> in the future direction of the
> * index scan, so they all get filled into a hash table.
> The MAX_MEMORY_PER_TABLE
> * property determines max cap of hash table 100. So from
> row 103 it goes into virtual
> * memory heap, whose in memory part is also 100 entries.
> So row 103 to 202 goes into
> * the in-memory part and gets dumped out in reverse
> order. Finally Row 203 to 250"
> * goes into file system. Here we mean row ids.
> */
>
> This test seems to be very sensitive to the precise query execution
> strategy that is being used, but I don't see how the test is
> controlling that query execution strategy.
>
> Can somebody clarify how the test works for me?
>
> thanks,
>
> bryan
>
> P.S. The reason that I'm asking is that my proposed change for
> DERBY-2526 causes this test to fail, but I'm not sure whether that
> means my change is bad, or whether this test is depending on a
> query execution strategy that is not guaranteed to occur.
>
>
>