[jira] [Commented] (DERBY-6888) Large User-Defined Types break Network Client

["Bryan Pendleton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DERBY-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269995#comment-15269995 ] 

Bryan Pendleton commented on DERBY-6888:
----------------------------------------

Hi Steven, thanks for the enormously useful and detailed analysis! I think your
writeup is very clear and accurate, and I agree with your assessment that
DERBY-4491 is intimately related to this problem.

A long, long time ago (10 years!) I worked on DERBY-125, which also involved
the transmission of large objects over the Derby client-server protocols.

So I can tell you, from experience, that: (a) it is certainly possible to use DRDA
protocols to send objects of arbitrary sizes, but (b) it is a pretty complex
undertaking to dig deeply into the protocols to the extent necessary.

Perhaps, the documents and links from DERBY-125 will give you some
background material to think about.

And, the fact that you've isolated some reproduction programs to demonstrate
the problems is tremendously valuable.

Are you interested enough in this problem to contemplate (with the
assistance of the community) trying to make an enhancement to 
Derby to remove this limitation?

> Large User-Defined Types break Network Client
> ---------------------------------------------
>
>                 Key: DERBY-6888
>                 URL: https://issues.apache.org/jira/browse/DERBY-6888
>             Project: Derby
>          Issue Type: Bug
>    Affects Versions: 10.11.1.1, 10.12.1.1
>            Reporter: Steven Schaefer
>         Attachments: udt_network_repro.zip
>
>
> In the project I work on at Nexidia we have been working with a derby in probably an unusual way where we make use of both the embedded and network clients.
> We also make some use of user-defined types and have run into some inconsistencies between those clients.
> Occasionally some of our user-defined types are above 32k. This is ok for the embedded client; it seems to read and write those without issue. However, the network client has some serious trouble.
> 1) Writes: The network client doesn't allow writing UDTs above 32k. I get this helpful error message (more on this at the end...) "java.sql.SQLDataException: The resulting value is outside the range for the data type 32767." Stack trace ends up pointing to org.apache.derby.client.net.Request.writeUDT, and its code shows a hardcoded check.
> 2) Reads: Since our (multi-server) system runs with both clients for different needs, our embedded client helpfully inserts UDTs greater than 32k. The network client doesn't appreciate this much. Reading one of these large UDTs results in the following: "java.sql.SQLNonTransientConnectionException: Insufficient data while reading from the network - expected a minimum of 6 bytes and received only 0 bytes.  The connection has been terminated".
> Even better, the next time we try to create a new connection, it fails with "java.sql.SQLNonTransientConnectionException: A network protocol error was encountered and the connection has been terminated: A protocol error (Data Stream Syntax Error) was detected.  Reason: 0x19. Perhaps this is an attempt to open a plaintext connection to an SSL enabled server."
> 3) With debug jars, #1's error message format fails an assertion check: {quote}
> org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED Number of parameters expected for message id 22003 (1) does not match number of arguments received (2)
> 	at org.apache.derby.shared.common.sanity.SanityManager.ASSERT(SanityManager.java:120)
> 	at org.apache.derby.shared.common.i18n.MessageUtil.formatMessage(MessageUtil.java:209)
> 	at org.apache.derby.shared.common.i18n.MessageUtil.getCompleteMessage(MessageUtil.java:118)
> 	at org.apache.derby.shared.common.i18n.MessageUtil.getCompleteMessage(MessageUtil.java:158)
> 	at org.apache.derby.shared.common.i18n.MessageUtil.getCompleteMessage(MessageUtil.java:74)
> 	at org.apache.derby.client.am.SqlException.<init>(SqlException.java:169)
> 	at org.apache.derby.client.am.SqlException.<init>(SqlException.java:203)
> 	at org.apache.derby.client.net.Request.writeUDT(Request.java:1418)
> {quote}    
>     
> h5. Expected
> I expect some consistency here between the two clients. Either both accept UDTs greater than 32k, or both reject them. Further, if a limit of 32k is still intended, it'd be great of documentation was a bit loud about that limitation. DERBY-4491 comments indicate some sort of "large UDT" may be appropriate, but it's not immediately clear to me if that's still needed. And of course I expect subsequent connections to succeed! I'm also a little surprised large blobs seem to work ok in this scenario, but I'm hoping that's an indication UDTs shouldn't be limited to 32k too.
> We've been able to refactor out the network client to workaround the issue, but it's not yet clear if this is an ideal long-term solution for us.
> h5. Reproducing:
> I have attached several sample programs:
> * *EmbeddedTest*: Inserts & Selects a blob and UDT that's 50k. I expect this program to run without issue.
> * *NetworkWriteTest*: Attempts to insert a blob and UDT that's 50k. This will demonstrate blobs are ok here, but #1 for UDTs (or #3 with debug jars).
> * *NetworkLargeUDTReadTest*: This demonstrates #2. It will attempt to read the large UDT written by EmbeddedTest. Finally, it will then try to create a new connection demonstrating the 0x19 error.
> h5. To run:
> * unzip, then cd udt_network_repo
> * gradlew build
> * java -cp "build\libs\*" com.nexidia.udtrepro.EmbeddedTest \[preferred derby home\]
> * java -cp "build\libs\*" com.nexidia.udtrepro.NetworkWriteTest \[preferred derby home\]
> * java -cp "build\libs\*" com.nexidia.udtrepro.NetworkLargeUDTReadTest \[preferred derby home\]
> Note: EmbeddedTest should be run before NetworkLargeUDTReadTest
> I have reproed all with 10.12 as well. All tests run in Windows 7 w/ Java 8 u77.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)