[jira] [Commented] (TIKA-4793) Make the Pipes IPC max payload size configurable (currently hard-coded to 100 MB)

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104077#comment-18104077 ] 

ASF GitHub Bot commented on TIKA-4793:
--------------------------------------

srujana-kuntumalla opened a new pull request, #3009:
URL: https://github.com/apache/tika/pull/3009

   ## Summary
   
   Follow-on to the earlier TIKA-4793 configurable-payload-limit work. Four issues were found after that PR:
   
   - **OOM in forked JVM for large documents** (e.g. 500 MB text with uncapped `maxIpcPayloadBytes`): `ServerProtocolIO.writeFinished()` called `JsonPipesIpc.toBytes()` unconditionally, allocating a full Smile `byte[]` in one shot. For large content this exhausted heap.
   - **Heap-bound error on very large single files**: Same root cause — Jackson's `ByteArrayBuilder` cannot produce a `byte[]` > 2 GB, so content that serializes beyond that limit caused `OutOfMemoryError` inside `toBytes()`.
   - **PDF "truncation" past ~80 M chars**: For Unicode-heavy PDFs, 80 M chars × 3 bytes/char UTF-8 Smile exceeds the 100 MB default limit. Previously the server wrote the oversized payload anyway, the client threw `PayloadLimitExceededException`, the IPC stream desynced, and the connection was torn down. Users saw "no content returned" and interpreted it as truncation.
   - **7z / rar / iso DYNAMIC emit-strategy sizing**: `EmitDataImpl.estimateSizeInBytes()` used `length × 2` (Java UTF-16 heap cost), but the DYNAMIC threshold is in wire bytes (Smile UTF-8). ASCII-heavy archive metadata (file paths, checksums, MIME types) encodes at ~1 byte/char, so estimates were 2× too high and compressed-archive results were incorrectly routed to direct-emit.
   
   ## Changes
   
   **`ServerProtocolIO`** — three-layer guard in `writeFinished()`:
   1. **Pre-check**: if `getEstimatedSizeBytes() > maxPayloadBytes`, skip serialization entirely (prevents OOM before any allocation).
   2. **OOM catch**: wraps `JsonPipesIpc.toBytes()` — caught `OutOfMemoryError` frees the byte-builder segments; the tiny `PAYLOAD_LIMIT_EXCEEDED` response then serializes cleanly.
   3. **Post-check**: if serialized `bytes.length > maxPayloadBytes` (CJK-heavy content where the 1 byte/char pre-estimate is optimistic), discard server-side — client sees a clean `PAYLOAD_LIMIT_EXCEEDED`, no stream desync.
   
   Constructor now takes `maxPayloadBytes` explicitly.
   
   **`PipesServer` / `ConnectionHandler`** — pass `pipesConfig.getMaxIpcPayloadBytes()` to `ServerProtocolIO` (one line each).
   
   **`EmitDataImpl.estimateSizeInBytes()`** — change multiplier from `length × 2` to `length` (1 byte/char ≈ Smile UTF-8 ASCII). The server-side post-check is the safety net for content that does serialize larger than the estimate.
   
   **`ServerProtocolIOTest`** (new) — unit tests for all three protection layers plus a pin on the corrected estimate formula.
   
   ## Test plan
   
   - [ ] `ServerProtocolIOTest` — 5 new tests covering pre-check, post-check, OOM path, status-only pass-through, and estimate formula
   - [ ] `PipesMessageTest` — 17 existing wire-protocol tests still pass
   - [ ] No existing integration test behavior changes (all test documents are well below the DYNAMIC strategy thresholds; UNPACK and CONTENT_ONLY modes bypass the threshold entirely)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)




> Make the Pipes IPC max payload size configurable (currently hard-coded to 100 MB)
> ---------------------------------------------------------------------------------
>
>                 Key: TIKA-4793
>                 URL: https://issues.apache.org/jira/browse/TIKA-4793
>             Project: Tika
>          Issue Type: Improvement
>          Components: tika-pipes
>            Reporter: Srinivasarao Daruna
>            Priority: Major
>
> PipesMessage.MAX_PAYLOAD_BYTES (tika-pipes-core) is a compile-time constant set to 100 MB:
> // tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/protocol/PipesMessage.java:44
> public static final int MAX_PAYLOAD_BYTES = 100 * 1024 * 1024;
> This cap is enforced on the read side of every IPC message in the PipesClient↔PipesServer socket protocol. It does not limit the file size being parsed (files are fetched server-side by a Fetcher); it limits the size of the serialized JSON payload — most critically the PipesResult (parsed metadata + extracted text) returned in FINISHED messages.
> Problems with the current implementation:
> 1. Hard-coded, not configurable. Users with very large documents that produce large parse results (and no MetadataWriteLimiterFactory configured) have no way to raise the cap short of forking the code. There is no corresponding field in PipesConfig.
> 2. No write-side guard. PipesMessage.write() applies no limit before writing. When the server serializes a PipesResult exceeding 100 MB and sends it, the client's PipesMessage.read() throws IOException("Payload length X exceeds maximum of 104857600 bytes"). This is caught by the catch-all Exception block in PipesClient.waitForServer() and surfaced to the caller as UNSPECIFIED_CRASH — a misleading status that provides no indication of the root cause.
> Proposed fix:
> 1. Add maxIpcPayloadBytes to PipesConfig with a default of 100 * 1024 * 1024, loaded from the "pipes" JSON config section (consistent with all other PipesConfig fields).
> 2. Thread the configured value through to both PipesMessage.read() and PipesMessage.write(), replacing the hard-coded constant.
> 3. Add a write-side guard in PipesMessage.write() so oversized results are caught server-side with a descriptive IOException rather than failing silently at the client with UNSPECIFIED_CRASH.
> Example config (proposed):
> {
>   "pipes": {
>     "maxIpcPayloadBytes": 209715200
>   }
> }
> Note: Users hitting this limit should first consider configuring a MetadataWriteLimiterFactory to bound extracted-text size, which is the right long-term solution for very large documents. But the limit should still be configurable for cases where the full content is legitimately needed.
> Affected files:
> - tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/protocol/PipesMessage.java
> - tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/PipesConfig.java
> - tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/PipesClient.java
> - tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)