You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Steve Lawrence (JIRA)" <ji...@apache.org> on 2018/01/26 20:09:06 UTC
[jira] [Updated] (DAFFODIL-1799) Enable data streaming in the CLI
[ https://issues.apache.org/jira/browse/DAFFODIL-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Lawrence updated DAFFODIL-1799:
-------------------------------------
Fix Version/s: (was: 2.1.0)
2.2.0
2.2.0
> Enable data streaming in the CLI
> --------------------------------
>
> Key: DAFFODIL-1799
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1799
> Project: Daffodil
> Issue Type: Bug
> Components: CLI, Performance
> Reporter: Steve Lawrence
> Priority: Major
> Fix For: 2.2.0
>
>
> This CLI currently reads the input data as a byte array. This is simple and allows for ensuring all data is read into a memory, reducing disk overhead during the preformance command. However, this means the CLI is limited to the maximum size of an array, which is INT_MAX. In order to support the CLI parsing/unparsing larger files, we should instead work on InputStreams rather than array buffers. For the performance subcommand, this will mean requiring something like a SplittalbeInputStream that will allow multiple consumers of a single InputStream.
> Some SplittableInputStream implementations do exist, for example in JMRTD and on stack overflow, but licensing issues make it so these aren't an option. Either need to find a solution compatible with our license or implement our own.
> This work should be done concurrently with changes to improve the efficiency of the I/O layer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)