[jira] Closed: (DERBY-867) Performance issues with large object handling in DDMWriter

["Bryan Pendleton (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DERBY-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Pendleton closed DERBY-867.
---------------------------------

    Resolution: Won't Fix

I agree with the observation that we can close this.

> Performance issues with large object handling in DDMWriter
> ----------------------------------------------------------
>
>                 Key: DERBY-867
>                 URL: https://issues.apache.org/jira/browse/DERBY-867
>             Project: Derby
>          Issue Type: Improvement
>          Components: Network Server, Performance
>    Affects Versions: 10.1.1.0, 10.2.1.6
>            Reporter: Bryan Pendleton
>            Priority: Minor
>
> There are some performance issues related to the way that DDMWriter handles extremely large network messages, such as those that arise in DERBY-125 and DERBY-492. The performance problems involve both memory usage and CPU cycles.
> Among the problems that could be investigated and improved:
> 1) The DDMWriter internal byte array grows, but never shrinks (except when the DDMWriter as a whole is collected). This means that there may be some memory wastage when the buffer grows to a large value, then is later used for only small buffers.
> 2) When the buffer grows, it tends to double in size. When you start gettting to moderately large buffer sizes, this may mean that the  buffer over-expands during a growth event, and grows to a larger size than is needed, again leading to memory wastage.
> 3) I'm not sure what would happen if memory were unavailable when the buffer wanted to grow; I think it would be a very hard error. There may be ways to handle this better, such as: waiting a while to see if memory becomes available; writing previously chained messages out of the buffer to free up space, etc. At the very least, it would be nice if an out-of-memory problem were handled cleanly, for example by terminating this connection but leaving the rest of the Network Server online and available for other users.
> 4) Perhaps we could eliminate the expansion all together by instead of expanding have a pool of buffers available to all the DDMWriters.  If  a DDMWriter needs to write more,  it picks up a new buffer and returns it to the pool when no longer needed.
> 5) Large DDM objects which require Layer B segmentation are written as a single object, then later the segmentation is added after the payload data is complete. This processing is handled by endDdm() and finalizeDssLength(). This code currently copies the bytes around in the buffer two "extra" times: once to insert the extended length bytes at the front of the large DDM object, and a second time to insert the DSS Continuation Headers every 32K bytes. Given the API between DDMWriter and its callers, it may be impossible to totally remove this memory copying, but it does seem like we could reduce from 2 full copies to 1, by combining the insertion of the extended length bytes with the continuation header segmentation. It might also be possible to detect a large object once it spills out from one DSS block into the second block, and hence avoid copying for subsequent blocks, even if those first two blocks do have to be copied once.
> There may be other opportunities in this part of the code; this issue is just a collection of the ideas that came up while the developers were discussing bugs DERBY-125, DERBY-170, DERBY-491, and DERBY-492. A review of the processing that occurs in those test cases should be performed as part of working on this bug.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.