You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Andreas Korneliussen (JIRA)" <de...@db.apache.org> on 2006/07/21 14:07:13 UTC
[jira] Created: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
----------------------------------------------------------------------------------------------------------------------------------------
Key: DERBY-1559
URL: http://issues.apache.org/jira/browse/DERBY-1559
Project: Derby
Issue Type: Sub-task
Components: Network Server
Affects Versions: 10.2.0.0
Reporter: Andreas Korneliussen
Assigned To: Andreas Korneliussen
When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
The blob data is then inserted into the DB by using
PreparedStatement.setBytes(..)
and later
PreparedStatement.execute()
To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
--
CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
--
Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
--
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12431227 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
Thanks for the review.
I will run some tests, and then commit the patch on the trunk.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Fix Version/s: 10.3.0.0
Derby Info: (was: [Patch Available])
Committed revision 438478 at trunk.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Fix For: 10.3.0.0
>
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, DERBY-1559v6.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Bryan Pendleton (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430363 ]
Bryan Pendleton commented on DERBY-1559:
----------------------------------------
Thanks Andreas. I fear I was perhaps a little vague in my previous comment.
The code I was concerned about was the code in EXTDTAReaderInputStream.java that looks like:
} catch (DRDAProtocolException e) {
throw new IOException(e.getMessage());
}
which is a pattern that occurs in several places in that class.
I was concerned that all the information we were extracting from the exception object "e"
was the getMessage() string, and were hence losing the stack trace of where exactly
the original DRDAProtocolException was thrown.
I was suggesting that we might change that code into:
} catch (DRDAProtocolException e) {
throw new IOException(e.getMessage(), e);
}
so that exception 'e' would be recorded as the "cause" of the new IOException,
but this would require using the new JDK 1.4 version of the constructor, so
instead we might try to figure out some other way to capture the stack trace
information from the exception "e".
Does my comment make more sense now?
thanks,
bryan
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Urgency: Normal
Affects Version/s: 10.2.2.0
10.3.0.0
Derby Info: [Patch Available]
I will consider committing the patch next week, unless anyone objects.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559v4.diff
Thanks for reviewing the patch. Attached is a patch where I try to address the issue w.r.t preserving the information in the DRDAProtocolException, if thrown during streaming. The DRDAProtocolException will be logged from the DDMReader, which has access to the DRDAConnThread, before being thrown again.
I have also considered some other options, like making DRDAProtocolException inherit from IOException or make a new IOException subclass, which is able to preserve the stack trace from the cause (in JDK 1.3). I did not do that, since
1. by making DRDAProtocolException inherit from IOException, I would proably need to go through all code and check for catching of IOException (which then would also catch DRDAProtocolException)
2. since we probably soon will stop supporting 1.3, I did not create a new IOException subclass.
(ironically DRDAProtocolException seems to usually be thrown as a consequence of an IOException)
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429384 ]
Tomohito Nakayama commented on DERBY-1559:
------------------------------------------
Hello, Andreas.
Discussion in DERBY-1610 resulted to the answer that we need to solve DERBY-1610
before DERBY-1559 and DERBY-1535.
DERBY-1559 and DERBY-1535 will cause difference in compatibility of types and
DERBY-1610, which is the issue around difference in compatibility of types between Network and Embedded ,is required to be solved in order to accept that difference.
Now it is not completely decided extent of modification in DERBY-1610 and I will post my plan in soon.
If you have some opinion for DERBY-1610, please comment it.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Bryan Pendleton (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430516 ]
Bryan Pendleton commented on DERBY-1559:
----------------------------------------
Yes, I understand now. I was just looking at the new class in a static fashion, and not
really understanding where the DRDAProtocolException was coming from. Thanks for the clarification.
Now your other comment
> (ironically DRDAProtocolException seems to usually be thrown as a consequence of an IOException)
makes a lot more sense, too.
One last thought, then: does it make any sense to have DDMReader.readLobContinuationString
throw an IOException rather than a DRDAProtocolException?
Then you wouldn't need any try/catch blocks in EXTDTAReaderInputStream at all, but could just
allow the IOExceptions to escape upward to wherever they will be caught higher up in the stack.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429394 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
DERBY-1535 caused conflicts with this patch during development. Now, DERBY-1610 will cause conflicts with it. I think this patch is bigger, and more complex than the patches in DERBY-1610, so it is probably better to commit this patch first, then update, resolve conflicts, and commit the patch in DERBY-1610.
Is that ok ?
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430431 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
Hi,
I thought I had addressed the comment by capturing the stack trace in DDMReader.readLobContiniationStream(). That is the only call that EXTDTAReaderInputStream makes, which can throw DRDAProtocolException.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Bryan Pendleton (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430147 ]
Bryan Pendleton commented on DERBY-1559:
----------------------------------------
Hi Andreas, I read your patch and I think it looks quite good.
I worry about potentially losing valuable information from the DRDAProtocolException
instance when we catch it and transform it into an IOException.
Two possible ideas:
1) Use initCause() to embed the DRDAProtocolException into the IOException,
although maybe that's off limits because it's a 1.4 feature?
2) Before throwing the IOException, inside the catch block for the DRDAProtocolException,
log the full DRDAProtocolException somewhere. This might mean that when you
construct an EXTDTAReaderInputStream instance, you pass it some sort of
callback hook that gives it access to the Network Server's error log / console.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429735 ]
Tomohito Nakayama commented on DERBY-1559:
------------------------------------------
I rollbacked DERBY-1535 and DERBY-1513.
I think it will make a conflict on this issue.
Please take your attention for it.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430034 ]
Tomohito Nakayama commented on DERBY-1559:
------------------------------------------
I think of committing your v3 patch and filing new issue that there remains room to improve memory usage,
only if we finish DERBY-1610 and remove use of setBytes.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen closed DERBY-1559.
---------------------------------------
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Fix For: 10.3.0.0, 10.2.2.0
>
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, DERBY-1559v6.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559v3.diff
Attached is an updated patch were the conflicts from DERBY-1610 has been resolved.
Additionally, to avoid the failures in jdbcapi/parameterMapping, I have modified the code so that it does only use setBinaryStream if it is a BLOB columns, otherwise it continues to use setBytes.
I have also done some memory profiling, were the code inserts one BLOB of size 64 MB streamed from the network client.
The results are:
Without any changes: max memory usage 350 MB
With a special patch were setBytes was replaced by setBinaryStream: 176 MB
With this patch: 40 MB.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12423868 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
Thanks
I will update the patch to follow the approach suggested in DERBY-1535 when receiving multiple EXTDTA.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.0.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429403 ]
Tomohito Nakayama commented on DERBY-1559:
------------------------------------------
Well.....
If *you* don't think the difference of DerbyNetClient/parameterMapping.out is critically harmful,
which was introduced in DERBY-1535 before, I think it is OK for you to commit DERBY-1559.
It is your judgement....
The background is that
DERBY-1559 also would require the difference of DerbyNetClient/parameterMapping.out and
committing DERBY-1559 will strengthen the difference.
If you think difference of DerbyNetClient/parameterMapping.out is critically harmful,
please do not commit patch of DERBY-1559 and wait for DERBY-1610.
Otherwise you would make unexpected reason for a problem against your will.
However opinion for that difference may differ betweeen people...
Then I left judge for you.
At least for me, through discussion in DERBY-1610, think already applied difference in DerbyNetClient/parameterMapping.out is unfinished and much more difference is needed.
and think it should be rollbacked and applied in once if possible.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello.
Thank you for your carefulness.
Please give your watch on DERBY-1610 and decide when you commit DERBY-1559.
Best regards.
Andreas Korneliussen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429406 ]
>
>Andreas Korneliussen commented on DERBY-1559:
>---------------------------------------------
>
>I will wait for DERBY-1610 being fixed, to avoid the risk.
>
>
>
>
>>when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
>>----------------------------------------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-1559
>> URL: http://issues.apache.org/jira/browse/DERBY-1559
>> Project: Derby
>> Issue Type: Sub-task
>> Components: Network Server
>> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
>> Reporter: Andreas Korneliussen
>> Assigned To: Andreas Korneliussen
>> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>>
>>
>>When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
>>The blob data is then inserted into the DB by using
>>PreparedStatement.setBytes(..)
>>and later
>>PreparedStatement.execute()
>>To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
>>I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
>>--
>>CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
>>--
>>Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
>>--
>>
>>
>
>
>
--
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
tmnk@apache.org
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12429406 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
I will wait for DERBY-1610 being fixed, to avoid the risk.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Bryan Pendleton (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430983 ]
Bryan Pendleton commented on DERBY-1559:
----------------------------------------
Thanks Andreas! The v5 patch fully addresses my concern about the exception
signatures in EXTDTAReaderInputStream. +1 to commit.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen resolved DERBY-1559.
-----------------------------------------
Fix Version/s: 10.2.2.0
Resolution: Fixed
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Fix For: 10.2.2.0, 10.3.0.0
>
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, DERBY-1559v6.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559.diff
DERBY-1559.stat
Attached is a patch which does the following:
EXTDTAReaderInputStream: this is a new class which is a subclass of InputStream. It is capable of reading EXTDTA from the network using the DDMReader.
DDMReader: added helper methods to create EXTDTAReaderInputStream and to let it fetch more data
DRDAConnThread:
* When handling a EXCSQLSTT request DRDAConnThread will do an execute of the statement right after calling readAndSetAllExtParams(..). This is because the call to stmt.execute() will start the stream of the LOB data. readAndSetExtParam (..) is implemented so that it will only do streaming of LOB on the last EXTDTA parameter, otherwise it will use the old mechanism of streaming all and creating a byte-array.
Using this patch, I have successfully inserted a 1GB blob streamed from the Network client to the Network server, running the Network server with 64MB of max heap space. I have also run derbyall with this patch, with no failures.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.0.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Tomohito Nakayama updated DERBY-1559:
-------------------------------------
Attachment: serverMemoryUsage.xls
I measured the execution of BlobOutOfMem.java with the patch of DERBY-1559v2.diff.
I attach the chart of it.
As the chart tolds,memory less than 64m was needed in Network Server to execute BlobOutOfMmem.java with the patch.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.0.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559v6.diff
Found a bug while testing the patch DERBY-1559v5.diff:
In case of an DRDAProtoclException while doing the streaming of the data, the DDMReader.readLOBContinuationStream(.) method called agent.handleException(..). This will cause the connection to be rolled back from within the stream (within an execute statement), and the engine throws an exception:
Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = /export/home/tmp/db/bigdb2;create=true; diagnostic msg = Cannot issue rollback in a nested connection when there is a pending operation in the parent connection.
org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = /exp ort/home/tmp/db/bigdb2;create=true; diagnostic msg = Cannot issue rollback in a nested connection when there is a pending opera tion in the parent connection.
The side-effect of this error is quite severe, since it seems that the connection will never be rolled back.
The attached patch DERBY-1559v6.diff addresses this by having DDMReader.readLOBContinuationStream(.) only log the exception. The connection will be rolled back later by DRDAConnectionThread when the exception comes out from statement.execute().
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, DERBY-1559v6.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12430519 ]
Andreas Korneliussen commented on DERBY-1559:
---------------------------------------------
> One last thought, then: does it make any sense to have DDMReader.readLobContinuationString
> throw an IOException rather than a DRDAProtocolException?
>
> Then you wouldn't need any try/catch blocks in EXTDTAReaderInputStream at all, but could just
> allow the IOExceptions to escape upward to wherever they will be caught higher up in the stack.
>
Yes, that makes sense. I will try that out.
Thanks.,
Andreas
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1559) when receiving a single EXTDTA
object representing a BLOB, the server do not need to read it into memory
before inserting it into the DB
Posted by "Tomohito Nakayama (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=comments#action_12423866 ]
Tomohito Nakayama commented on DERBY-1559:
------------------------------------------
In DERBY-1535, it was suggested that Engine itself takes more amount of memory in the case ps.setBytes was called than in the case ps.setBinaryStream was called.
Then, there are room to improve memory usage in multiple EXDTA, just by calling ps.setBinaryStream with ByteArrayInputStream object which interfaces byte[] paramBytes, instead of ps.setBytes.
However, interfacing byte array via ByteArrayInputStream is a temporary measure before multiple EXTDTA are streamed. Then, multiple EXDTA streaming is prefered, if we could achieve it ...
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.0.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559v2.diff
Attached is an updated patch which preserves the changes in DERBY-1535 when receiving multiple EXTDTA objects.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.0.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1559) when receiving a single EXTDTA object
representing a BLOB, the server do not need to read it into memory before
inserting it into the DB
Posted by "Andreas Korneliussen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1559?page=all ]
Andreas Korneliussen updated DERBY-1559:
----------------------------------------
Attachment: DERBY-1559v5.diff
Attached patch makes DDMReader handle the DRDAProtocolException, and throws IOException from it, instead of doing it from the stream class.
> when receiving a single EXTDTA object representing a BLOB, the server do not need to read it into memory before inserting it into the DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1559
> URL: http://issues.apache.org/jira/browse/DERBY-1559
> Project: Derby
> Issue Type: Sub-task
> Components: Network Server
> Affects Versions: 10.2.1.0, 10.3.0.0, 10.2.2.0
> Reporter: Andreas Korneliussen
> Assigned To: Andreas Korneliussen
> Attachments: DERBY-1559.diff, DERBY-1559.stat, DERBY-1559v2.diff, DERBY-1559v3.diff, DERBY-1559v4.diff, DERBY-1559v5.diff, serverMemoryUsage.xls
>
>
> When streaming a BLOB from the Network Client to the Network Server, the Network server currently read all the data from the stream and put it into a byte array.
> The blob data is then inserted into the DB by using
> PreparedStatement.setBytes(..)
> and later
> PreparedStatement.execute()
> To avoid OutOfMemoryError if the size of the Blob is > than total memory in the VM, we could make the network server create a stream which reads data when doing PreparedStatement.execute(). The DB will then stream the BLOB data directly from the network inputstream into the disk.
> I intent to make a patch which does this if there is only one EXTDTA object (BLOB) sent from the client in the statement, as it will simplify the implementation. Later this can be improved further to include CLOBs, and possibly to include the cases where there are multiple EXTDTA objects.
> --
> CLOBs are more complex, as there need to be some character encoding. This can be achieved by using a InputStreamReader, and use PreparedStatement.setCharacterStream(..). However the size of the stream is not necessarily the same as the size of the raw binary data, and to do this for CLOBs, I would need the embedded prepared statements to support the new setCharacterStream() overloads in JDBC4 (which do not include a length atribute)
> --
> Multiple EXTDATA objects are also more complex, since one would need to have fully read the previous object ,before it is possible to read the next.
> --
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira