[jira] Commented: (DERBY-352) Clobs on insert using streams should not instantiate into memory

["Sunitha Kambhampati (JIRA)" <de...@db.apache.org>]

    [ http://issues.apache.org/jira/browse/DERBY-352?page=comments#action_12331304 ] 

Sunitha Kambhampati commented on DERBY-352:
-------------------------------------------

Problem:
if you insert a clob value using streams, the stream is materialized as part of the normalization process and the store layer never sees the stream but instead finds a char array. ReaderToUTF8Stream converts the character from the stream to cloudscape/derby UTF8 encoding which is how characters are stored in derby.  The store layer ideally should drive the streaming and store already handles the streaming via RememberBytesInputStream ( see StoredPage.logColumn).

This patch
1. adds changes to not do materialization of the stream at normalization 
	- fix bug in ReaderToUTF8Stream.read(b[],,) to allow for correct reads
	- changes to SQLClob to not materialize stream
	- per sql standard, truncation of clobs is allowed for trailing blanks and this is the behavior of clobs in 
	  derby today.  Keep information about truncation information along with the stream. The truncation checking logic
	  is pushed to the stream(ReaderToUTF8Stream) and thus will be enforced only when the stream is read.
	- With these changes, the stream will be read only once, driven by store layer. This is efficient. 
	  Earlier, the stream would be converted to char array  which involves the char to derby specific utf8 encoding, and then back from utf8encoding to char as part of the materialization and once again at the store layer, the char is converted to derby specific utf8encoding.  

Also note, no changes have been made to existing behavior of streams for char,varchar and long varchar. This fix includes changes only for clobs.

2. Test changes
	- added new test for clobs in characterStreams.java
	- added new test for clob truncation in streamingColumn.java
	  Note, also made changes to disable logStatmentText=true (derby595 and also disabling streamTest5).
	- added tests to test for insert,select,update of 2G clob and blobs, as part of the largedata suite. This test LobLimits.java will not run as part of derbyall because it needs considerable amount of disk space and also takes long  time to run.I have added extra tests for blobs although it is not related to this fix and as a result have found some issues that will need to be looked into. In LobLimits.java, some tests are therefore currently disabled. 
	- there are existing stream tests that run as part of derbyall ( ex blobclob4BLOB.java,resultsetStream.java) that will test for clobs streaming but may not be able to catch if clobs are being materialized or not, which is why it is important to run the largedata/LobLimits at some intervals to ensure that no regression has occurred.


Ran derbyall on jdk142/win2k ok.

svn stat
M      java\engine\org\apache\derby\impl\jdbc\EmbedResultSet.java
M      java\engine\org\apache\derby\impl\jdbc\EmbedPreparedStatement.java
M      java\engine\org\apache\derby\impl\jdbc\ReaderToUTF8Stream.java
M      java\engine\org\apache\derby\iapi\types\SQLClob.java
A      java\testing\org\apache\derbyTesting\functionTests\tests\largedata\LobLimits_app.properties
M      java\testing\org\apache\derbyTesting\functionTests\tests\largedata\copyfiles.ant
A      java\testing\org\apache\derbyTesting\functionTests\tests\largedata\LobLimits.java
M      java\testing\org\apache\derbyTesting\functionTests\tests\jdbcapi\characterStreams.java
M      java\testing\org\apache\derbyTesting\functionTests\tests\store\streamingColumn.java
M      java\testing\org\apache\derbyTesting\functionTests\tests\store\streamingColumn_derby.properties
M      java\testing\org\apache\derbyTesting\functionTests\master\streamingColumn.out
A      java\testing\org\apache\derbyTesting\functionTests\master\LobLimits.out
M      java\testing\org\apache\derbyTesting\functionTests\master\characterStreams.out
M      java\testing\org\apache\derbyTesting\functionTests\suites\largeDataTests.runall


This is my first fix in this area and I would appreciate critical eyes on this. Thanks.

> Clobs on insert  using streams should not instantiate into memory
> -----------------------------------------------------------------
>
>          Key: DERBY-352
>          URL: http://issues.apache.org/jira/browse/DERBY-352
>      Project: Derby
>         Type: Bug
>   Components: JDBC, Performance, Store
>     Versions: 10.0.2.1, 10.0.2.0
>  Environment: all
>     Reporter: Sunitha Kambhampati
>     Assignee: Sunitha Kambhampati
>  Attachments: Derby352.diff.txt
>
> Currently in derby, for an insert on a clob using setCharacterStream what will happen is , the entire stream will be materialized into a char array and sent to store for the insert. 
> see  - SQLVarChar.normalize, where getString() is called and SQLChar.getString() will materiliaze the entire stream into a char array ( SQLChar.readExternal). 
> The store layer must actually drive the streaming. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira