Commit patch for DERBY-326 (Re: [jira] Updated: (DERBY-326) Improve streaming of large objects for network server and client)

[TomohitoNakayama <to...@basil.ocn.ne.jp>]

Hello.

I will commit this patch, seeing no one point out anything in days ...

Best regards.

Tomohito Nakayama (JIRA) wrote:

>     [ http://issues.apache.org/jira/browse/DERBY-326?page=all ]
>
>Tomohito Nakayama updated DERBY-326:
>------------------------------------
>
>    Attachment: DERBY-326_7.patch
>
> I upload DERBY-326_7.patch. 
> --- 
>     Description of patch : 
>        * Remove processing to expand data from InputStream of blob/clob to memory before sending to client. 
>         * Implement layer B streaming at NetworkServer. 
>          * As written in this issue firstly, almost rewrite whole org.apache.derby.impl.drda.DDMWriter#writeScalarStream. 
>            Here , "almost" means that code was not wrote from scratch, but was wrote as reform. 
>            Remarkable point is as next : 
>            Original code was using variable "bytesToRead" to handle remaining amount of data sent and remaining roon in DSS segment . 
>            Now this variable "bytesToRead" was removed from. 
>            New code, instead, have variable "spareDssLength" to handle remaining room in DSS segment . 
>          * Add call to ensureLength in writeScalarStream expecting appropriate buffer size. 
>          * Move comment in java/drda/org/apache/derby/impl/drda/DDMWriter.java about client driver implementation 
>             to java/client/org/apache/derby/client/net/Request.java. 
>                 
>         * Modify org.apache.derby.impl.drda.EXTDTAInputStream to stream InputStream retrieved from ResultSet directly. 
>          * The source stream is read twice, first for seeing whether source stream is/is not empty, second for streaming it. 
>          * To keep reference to valid stream, EXTDTAInputStream have reference to resultset and blob/clob also. 
>               
>        * Modify master file of result for blobclob4BLOB. 
>         * Now as same as result of embed driver, dead lock will be happen in clobTest92. 
>         * Different expected exception was happen in negative test in blobclob4BLOB. 
>              
>        * Added asserting code. 
>        * Added negative test to kill streaming. 
> 
>        * Place buffer before lob object was streamed out to client.
>        * Added test to stream with the stream out buffer configuration.
>
>        * Other improvements from DERBY-326_6.patch were as next. 
>         * Remove the code for handling byte[].
>         * Make EXTDTAInputStream always be markSupported.
> 
>  Testing : 
>        * Executed derbyall and found no error except found in http://www.multinet.no/~solberg/public/Apache/DerbyJvm1.4/Limited/testSummary-396619.html
>
>
>  
>
>>Improve streaming of large objects for network server and client
>>----------------------------------------------------------------
>>
>>         Key: DERBY-326
>>         URL: http://issues.apache.org/jira/browse/DERBY-326
>>     Project: Derby
>>        Type: Improvement
>>    
>>
>
>  
>
>>  Components: Network Client, Performance, Network Server
>>    Reporter: Kathey Marsden
>>    Assignee: Tomohito Nakayama
>> Attachments: ClobTest.zip, DERBY-326.patch, DERBY-326_2.patch, DERBY-326_3.patch, DERBY-326_4.patch, DERBY-326_5.patch, DERBY-326_5_indented.patch, DERBY-326_6.patch, DERBY-326_7.patch, ReEncodedInputStream.java.modifiedForLongRun
>>
>>Currently the stream writing  methods in network server and client require a  length parameter. This means that we have to get the length of the stream before sending it. For example in network server in EXTDTAInputStream we have to use getString and getbytes() instead of getCharacterStream and getBinaryStream so that we can get the  length.
>>SQLAM Level 7 provides for the enhanced LOB processing to allow streaming without indicating the length, so, the writeScalarStream methods in
>>network server DDMWriter.java and network client Request.java can be changed to not require a length.
>>Code inspection of these methods seems to indicate that while the length is never written it is used heavily in generating the DSS. One strange thing is that it appears on error, the stream is padded out to full length with zeros, but an actual exception is never sent.  Basically I think perhaps these methods need to be rewritten from scratch based on the spec requirements for lobs.
>>After the writeScalarStream methods have been changed, then EXTDAInputStream can be changed to properly stream LOBS. See TODO tags in this file for more info.  I am guessing similar optimizations available in the client as well, but am not sure where that code is.
>>    
>>
>
>  
>

-- 
/*

        Tomohito Nakayama
        tomonaka@basil.ocn.ne.jp
        tomohito@rose.zero.ad.jp
        tmnk@apache.org

        Naka
        http://www5.ocn.ne.jp/~tomohito/TopPage.html

*/ 

Re: Commit patch for DERBY-326 (Re: [jira] Updated: (DERBY-326) Improve streaming of large objects for network server and client)

[Kathey Marsden <km...@sbcglobal.net>]

TomohitoNakayama wrote:

> Hello.
>
> I will commit this patch, seeing no one point out anything in days ...
>
Thank you Naka for your steadfast work on this important issue.  I have
been doing less and less in the  DRDA rhelm lately so won't be reviewing
but hope others get a chance to do that either before or after commit. 
Community reviews are  a key component to  success in an  open source
environment  I think.

Kathey