You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "RAVINARAYAN SINGH (Jira)" <ji...@apache.org> on 2025/09/28 01:57:00 UTC
[jira] [Created] (NIFI-15025) LookupFailureException with large HTTP responses due to BufferedInputStream mark/reset limitation
RAVINARAYAN SINGH created NIFI-15025:
----------------------------------------
Summary: LookupFailureException with large HTTP responses due to BufferedInputStream mark/reset limitation
Key: NIFI-15025
URL: https://issues.apache.org/jira/browse/NIFI-15025
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: RAVINARAYAN SINGH
Assignee: RAVINARAYAN SINGH
Fix For: 2.7.0
When wrapping an HTTP response body stream with BufferedInputStream for large files, NiFi record readers can fail with the following error:
{code:java}
org.apache.nifi.lookup.LookupFailureException: java.io.IOException: Resetting to invalid mark
Caused by: java.io.IOException: Resetting to invalid mark {code}
This happens because BufferedInputStream only supports mark/reset up to its internal buffer size (default 8 KB). Once the reader attempts to reset beyond that buffer, the stream becomes invalid.
Code from [RestLookupService.java| https://github.com/apache/nifi/blob/60330769f668abf963f8a32202841cafa10f1885/nifi-extension-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/RestLookupService.java#L383-L388]
{code:java}
final Record record;
try (final InputStream is = responseBody.byteStream();
final InputStream bufferedIn = new BufferedInputStream(is)) {
record = handleResponse(bufferedIn, responseBody.contentLength(), context);
} {code}
h3. *Proposed Fix / Solutions*
# {*}Remove BufferedInputStream{*}{*}{*}
Use responseBody.byteStream() directly if mark/reset is not required by the reader.
# {*}Configurable Buffer Size{*}{*}{*}
Introduce a NiFi property to configure the buffer size for streams that require buffering. Default could remain 8 KB, but users may increase it for larger payloads.
# {*}Spooling InputStream Wrapper (Recommended){*}{*}{*}
Provide a robust InputStream wrapper that preserves streaming while supporting unlimited mark/reset via spooling to disk:
** mark() records the current absolute position.
** reset() replays bytes starting from the marked position.
** Additional bytes beyond the replay window are streamed and spooled transparently.
** Temporary spool file is automatically deleted on stream close.
This ensures NiFi processors handle large HTTP payloads correctly without running into mark/reset limits or heap issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)