[jira] [Created] (DAFFODIL-2194) buffered data output stream has a chunk limit of 2GB

["Steve Lawrence (Jira)" <ji...@apache.org>]

Steve Lawrence created DAFFODIL-2194:
----------------------------------------

             Summary: buffered data output stream has a chunk limit of 2GB
                 Key: DAFFODIL-2194
                 URL: https://issues.apache.org/jira/browse/DAFFODIL-2194
             Project: Daffodil
          Issue Type: Bug
          Components: Back End
            Reporter: Steve Lawrence
             Fix For: 2.5.0


A buffered data outupt stream is backed by a growable ByteArrayOutputStream, which can only grow to 2GB in size. So if we ever try to write more than 2GB to a buffered output stream during unparse (very possible with large blobs), we'll get an OutOfMemoryError.

One potential solution is to be aware of the size of a ByteArrayOutputStream when buffering output and automatically create a split when it gets to 2GB in sizes. This will still require a ton of memory since we're buffering these in memoary, but we'll at least be able to unparse more than 2GB of continuous data. 

Note that we should still be able to unparse more than 2GB of data total, as long as there so single buffer that's more than 2GB.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)