[jira] [Commented] (NIFI-16322) GenerateFlowFile should stream File Size instead of allocating a byte array

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113017#comment-18113017 ] 

ASF subversion and git services commented on NIFI-16322:
--------------------------------------------------------

Commit 40752b9c6292996cd679cfa410bd864163da6d7c in nifi's branch refs/heads/NIFI-16322 from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=40752b9c629 ]

NIFI-16322 Stream GenerateFlowFile content instead of a full-size byte array

Co-authored-by: Cursor <cu...@cursor.com>


> GenerateFlowFile should stream File Size instead of allocating a byte array
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-16322
>                 URL: https://issues.apache.org/jira/browse/NIFI-16322
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Joe Witt
>            Assignee: Joe Witt
>            Priority: Major
>
> GenerateFlowFile builds the entire File Size in a heap byte[] before session.write.
> generateData() does asDataSize(DataUnit.B).intValue() then new byte[byteCount].
> Double.intValue() saturates at Integer.MAX_VALUE, so a legal value such as 3 GB is
> silently generated as 2,147,483,647 bytes. DATA_SIZE_VALIDATOR has no upper bound.
> The array is then written to the content repository, so the payload exists twice.
> Unique FlowFiles = false builds that array in @OnScheduled and holds it for the
> run. Unique FlowFiles = true allocates a new array per FlowFile in the batch.
> OutOfMemoryError is uncaught; the processor only has success.
> There is no protocol reason for a contiguous buffer. Generate random or text
> bytes into a small buffer and stream them through session.write until File Size
> bytes are written. Unique = false can reuse a seed or a modest buffer so FlowFiles
> still share the same content without caching a full-size array.
> GenerateFlowFile has no @SystemResourceConsideration(MEMORY) today. Do not add
> one that claims the whole File Size lives on heap. After streaming, leave the
> annotation off.
> Expected:
> - File Size is written in chunks via session.write; do not hold a byte[] of File Size.
> - Values greater than Integer.MAX_VALUE bytes are invalid or rejected without allocating.
> - Do not truncate 3 GB to 2 GiB - 1.
> - Unique FlowFiles and Batch Size behavior stay the same.
> - Custom Text is unchanged (File Size is already ignored on that path).
> - Small sizes (default 0B, existing tests) still produce the exact length.
> Tests (surefire TestGenerateFlowFile, no multi-GB fixture):
> - Size within cap produces a FlowFile of that exact length (unique and non-unique).
> - Size above Integer.MAX_VALUE is invalid at configuration or fails without
>   NegativeArraySizeException / a huge array.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)