You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by TomohitoNakayama <to...@basil.ocn.ne.jp> on 2006/01/31 15:53:13 UTC
Negative test case (Re: [jira] Commented: (DERBY-326) Improve streaming
of large objects for network server and client)
Hello Kathey.
Kathey Marsden (JIRA) wrote:
>In the email:
>http://mail-archives.apache.org/mod_mbox/db-derby-dev/200601.mbox/%3c43D01EA1.2000708@amberpoint.com%3e
>Bryan asked
>What sort of faults are we likely to hit? Are things
>like "corrupt page in database" the most likely?
>
>I don't know what is most likely, but I did think of one test case likely to cause an IOException.
>If the reader is in TRANSACTION_READ_UNCOMMITTED isolation mode and then another connection updated the LOB, the reader should get an IOException. on the next read.
>
Reading your comment above, I wrote test code attached to this mail, and
tried executing it with my latest patch, which is not submitted yet.
Then I found no Exception happens at all.
Are there any misunderstanding you in my test code ?
Or is this unlucky success ?
Best regards.
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
Re: Negative test case (Re: [jira] Commented: (DERBY-326) Improve
streaming of large objects for network server and client)
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello.
I have enlarged the size of lob data and try the test again.
The result was that update was failed because of dead lock.
This error is not around streaming and not what I expected to cause.
I came to think that it is difficult to make negative test situation for
streaming only with manipulating dbms via jdbc api.
// If it was possible, it may be just a bug to be fixed ....
Now I think to prepare debug code to cause exception while streaming
using SanityManager.
Best regards.
Bryan Pendleton wrote:
>>> If the reader is in TRANSACTION_READ_UNCOMMITTED isolation mode and
>>> then another connection updated the LOB, the reader should get an
>>> IOException. on the next read.
>>>
>> Reading your comment above, I wrote test code attached to this mail,
>> and tried executing it with my latest patch, which is not submitted yet.
>> Then I found no Exception happens at all.
>>
>> Are there any misunderstanding you in my test code ?
>> Or is this unlucky success ?
>
>
> I think your test program is good, but apparently it does not provoke
> the exception that Kathey has in mind.
>
> It occurs to me that, with your new code, lob data should be fetched
> from the server in chunks of approximately 32K, so I think that you
> may need to incorporate that information into your test program.
>
> That is, when the first connection goes back to read the second set of
> 256 bytes of blob data from the input stream, it might just be returning
> cached data from the first 32K segment that was returned from the server,
> and it might be that it won't encounter the server-side exception until
> it exhausts that first 32K segment and returns to the server for more
> data.
>
> What if you do something like this:
>
> 1) Initialize the blob column to contain a lot of data: e.g., 128K bytes
> 2) Have the first connection fetch the first 256 bytes, as you do now.
> 3) Have the second transaction update the blob to replace it with a
> very short value: e.g., 500 bytes of data total.
> 4) Then, have the first transaction attempt to fetch *all* the blob data.
>
> Here are some things that I think *might* happen at step 4:
> 1) The first transaction might get an "end-of-blob" after 500 bytes
> total.
> 2) The first transaction might get all 256K bytes of the original blob.
> 3) The first transaction might get 32K bytes of the original blob, then
> get an "end-of-blob"
> 4) The first transaction might get 32K bytes of the original blob, then
> get an IO Exception (I think this may be the behavior that Kathey was
> trying to expose).
>
> thanks,
>
> bryan
>
>
>
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
Re: Negative test case (Re: [jira] Commented: (DERBY-326) Improve
streaming of large objects for network server and client)
Posted by Bryan Pendleton <bp...@amberpoint.com>.
>> If the reader is in TRANSACTION_READ_UNCOMMITTED isolation mode and
>> then another connection updated the LOB, the reader should get an
>> IOException. on the next read.
>>
> Reading your comment above, I wrote test code attached to this mail, and
> tried executing it with my latest patch, which is not submitted yet.
> Then I found no Exception happens at all.
>
> Are there any misunderstanding you in my test code ?
> Or is this unlucky success ?
I think your test program is good, but apparently it does not provoke
the exception that Kathey has in mind.
It occurs to me that, with your new code, lob data should be fetched
from the server in chunks of approximately 32K, so I think that you
may need to incorporate that information into your test program.
That is, when the first connection goes back to read the second set of
256 bytes of blob data from the input stream, it might just be returning
cached data from the first 32K segment that was returned from the server,
and it might be that it won't encounter the server-side exception until
it exhausts that first 32K segment and returns to the server for more data.
What if you do something like this:
1) Initialize the blob column to contain a lot of data: e.g., 128K bytes
2) Have the first connection fetch the first 256 bytes, as you do now.
3) Have the second transaction update the blob to replace it with a
very short value: e.g., 500 bytes of data total.
4) Then, have the first transaction attempt to fetch *all* the blob data.
Here are some things that I think *might* happen at step 4:
1) The first transaction might get an "end-of-blob" after 500 bytes total.
2) The first transaction might get all 256K bytes of the original blob.
3) The first transaction might get 32K bytes of the original blob, then
get an "end-of-blob"
4) The first transaction might get 32K bytes of the original blob, then
get an IO Exception (I think this may be the behavior that Kathey was
trying to expose).
thanks,
bryan
Re: Negative test case (Re: [jira] Commented: (DERBY-326) Improve
streaming of large objects for network server and client)
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello, Bryan and Mike.
Thank you for your advises.
I will try more referring your comments ...
Best regards.
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
Re: Negative test case (Re: [jira] Commented: (DERBY-326) Improve
streaming of large objects for network server and client)
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I would have to do some research to tell exactly what happens in the
store, for read uncommitted. I believe there is no problem for other
isolation levels as we either keep locks, or copy the data on select.
Deletes just mark it deleted, data doesn't go away until commit. And
updates of multi-page blobs are not in place so even the same thread
updating the same blob it has opened in another thread is probably all
right. But all interesting test cases.
Note that I think it is impossible to guarantee an error
on the the next read. There are MANY possible layers of caching going
on and they depend on the following factors:
o embedded vs. client
o size of the lob vs. page size of the database (blobs less than a page
are going to be read into memory so deletes/updates won't affect).
o where in the stream you are positioned (from store, multi-page blobs
are read into memory a page at a time, so reads don't go back to
disk/cache until they get to the next page worth of the blob)
o whether the query had to materialize the blob for some reason (we
try hard not to, but there are cases that have to).
o activity of background space reclamation threads (currently deleted
rows are not reclaimed until all rows on the page are deleted, so
deleting a row with a blob may not remove the blob -- there are some
reported derby issues to increase the reclamation for rows with blobs
in them)
An interesting store centric test would create a table with big rows
such that there is one row per page (one way is to add a bunch of char
fields such that rows length is bigger than a page) and add a blob column.
create a blob that is bigger than the page, say something like 128k.
Then in embedded mode read the first byte as a stream. A simple select
of the row by primary key and fetch of the column as a stream should
let derby stream the blob out rather than materialize it.
Then as bryan suggests delete and commit the delete in another connection.
Then sleep for awhile to allow background reclamation of the deleted row.
And finally stream the rest of blob out.
I don't know the details of what kind of streaming is going on in the
network layer in blob streaming.
TomohitoNakayama wrote:
> Hello Kathey.
>
> Kathey Marsden (JIRA) wrote:
>
>> In the email:
>> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200601.mbox/%3c43D01EA1.2000708@amberpoint.com%3e
>>
>> Bryan asked
>> What sort of faults are we likely to hit? Are things
>> like "corrupt page in database" the most likely?
>>
>> I don't know what is most likely, but I did think of one test case
>> likely to cause an IOException.
>> If the reader is in TRANSACTION_READ_UNCOMMITTED isolation mode and
>> then another connection updated the LOB, the reader should get an
>> IOException. on the next read.
>>
> Reading your comment above, I wrote test code attached to this mail, and
> tried executing it with my latest patch, which is not submitted yet.
> Then I found no Exception happens at all.
>
> Are there any misunderstanding you in my test code ?
> Or is this unlucky success ?
>
> Best regards.
>
>
> ------------------------------------------------------------------------
>
> /*
>
> Derby - Class org.apache.derbyTesting.functionTests.tests.lang.DirtyReadOfLob
>
> Copyright 1999, 2005 The Apache Software Foundation or its licensors, as applicable.
>
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
>
> */
>
> package org.apache.derbyTesting.functionTests.tests.lang;
>
> import java.io.InputStream;
> import java.io.ByteArrayInputStream;
> import java.sql.Connection;
> import java.sql.Statement;
> import java.sql.PreparedStatement;
> import java.sql.ResultSet;
>
> import java.io.IOException;
> import java.sql.SQLException;
>
> import org.apache.derby.tools.ij;
>
> public class DirtyReadOfLob {
>
>
> public static void main(String[] args){
>
> try{
> ij.getPropertyArg(args);
>
>
> createTestTable();
>
> Connection conn1 = ij.startJBMS();
> Connection conn2 = ij.startJBMS();
>
> testDirtyReadOfLob( conn1,
> conn2 );
>
> conn1.close();
> conn2.close();
>
> }catch(Throwable t){
> t.printStackTrace();
>
> }
> }
>
>
> private static void createTestTable()
> throws SQLException,
> IllegalAccessException,
> ClassNotFoundException,
> InstantiationException
> {
>
> Connection conn = ij.startJBMS();
>
> Statement createTableSt = conn.createStatement();
> createTableSt.execute("create table TEST_TABLE( TEST_COL blob( 512 ))");
> createTableSt.close();
>
> PreparedStatement insertLobSt =
> conn.prepareStatement("insert into TEST_TABLE( TEST_COL ) values(?)");
>
> insertLobSt.setBinaryStream(1,createOriginalDataInputStream(),512);
> insertLobSt.executeUpdate();
>
> insertLobSt.close();
>
> conn.commit();
> conn.close();
>
> }
>
>
> private static void testDirtyReadOfLob(Connection conn1,
> Connection conn2)
> throws SQLException,
> IOException
> {
>
> conn1.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED);
> conn2.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED);
>
> conn1.setAutoCommit(false);
> conn2.setAutoCommit(false);
>
> PreparedStatement st = conn1.prepareStatement("select TEST_COL from TEST_TABLE");
> ResultSet rs = st.executeQuery();
> rs.next();
>
> InputStream is = rs.getBinaryStream(1);
>
> read256Bytes(is);
>
> updateLobValue(conn2);
> conn2.commit();
>
> read256Bytes(is);
>
> is.close();
> rs.close();
> st.close();
>
> conn1.commit();
>
> }
>
>
> private static ByteArrayInputStream createOriginalDataInputStream(){
>
> byte[] originalValue = new byte[ 512 ];
>
> for(int i = 0; i < originalValue.length; i ++){
> originalValue[i] = (byte) (i % 256);
> }
>
> return new ByteArrayInputStream(originalValue);
>
> }
>
>
> private static ByteArrayInputStream createUpdatedDataInputStream(){
>
> byte[] updatedValue = new byte[ 512 ];
>
> for(int i = 0; i < updatedValue.length; i ++){
> updatedValue[i] = (byte) ( (updatedValue.length - i) % 256 );
> }
>
> return new ByteArrayInputStream(updatedValue);
>
> }
>
>
> private static void read256Bytes(InputStream is) throws IOException{
>
> for(int i = 0;
> i < 256;
> i ++){
>
> System.out.print(is.read());
> System.out.print(',');
>
> }
>
> System.out.println();
>
> }
>
> private static void updateLobValue(Connection conn) throws SQLException{
>
> PreparedStatement updateSt = conn.prepareStatement("update TEST_TABLE set TEST_COL = ?");
> updateSt.setBinaryStream(1,createUpdatedDataInputStream(),512);
> updateSt.executeUpdate();
> updateSt.close();
> }
>
> }