[jira] [Commented] (TIKA-4795) ParseBytes: parse-only entrypoint for callers that already hold the document bytes

["Davide Polato (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110609#comment-18110609 ] 

Davide Polato commented on TIKA-4795:
-------------------------------------

Implemented on [TIKA-4795-parseBytes|https://github.com/ai-pipestream/tika/tree/TIKA-4795-parseBytes] : bounded unary on TikaV2, bytes routed like a tika-server body (inline up to pipes.maxInlineBytes, spooled above it), content cap and inbound limit as separate knobs, provenance echoed, fields 7-10 still reserved. Description updated with what got decided.
Tested end to end from StormCrawler: [parsebytes-probe|https://github.com/dpol1/parsebytes-probe], 15 checks on 15 URLs, archived run against that commit.

> ParseBytes: parse-only entrypoint for callers that already hold the document bytes
> ----------------------------------------------------------------------------------
>
>                 Key: TIKA-4795
>                 URL: https://issues.apache.org/jira/browse/TIKA-4795
>             Project: Tika
>          Issue Type: New Feature
>          Components: tika-pipes
>    Affects Versions: 4.0.0
>            Reporter: Davide Polato
>            Priority: Major
>              Labels: grpc, pipes, protobuf
>
> h4. Goal
> Add a parse-only tika-grpc entrypoint that parses the exact bytes supplied by the caller, without asking Tika to fetch or re-fetch the resource.
> This was split from the typed {{Document}} work in [TIKA-4766|https://issues.apache.org/jira/browse/TIKA-4766] / [PR #2961|https://github.com/apache/tika/pull/2961], whose description states that "{{ParseBytes}} is deferred to its own JIRA."
> h4. Motivation
> {{FetchAndParse}} obtains content through a registered fetcher. Callers such as web crawlers have already acquired the resource bytes. Requiring another acquisition path can duplicate the transfer or parse a different representation from the one the caller observed because of redirects, cookies, authentication, robots and rate-limiting decisions, or transient content.
> The parse result should correspond to the exact representation captured by the caller.
> Apache StormCrawler is a concrete consumer. Its parsing bolts already receive the acquired bytes together with URL and protocol metadata. StormCrawler can contribute crawler requirements and input fixtures including truncated payloads, declared-vs-actual charset mismatches, malformed archives, documents containing embedded resources, HTML with {{<base>}}, and distinct URLs with identical content.
> h4. Proposed observable behavior
> This issue defines observable behavior, not the internal buffering, fetcher, or Pipes implementation.
> The request carries the exact byte sequence. It may also carry:
> * an optional opaque caller correlation id, echoed but never interpreted
> * source URI and effective URI when available, plus a base URI for link resolution; these are provenance metadata and must never be dereferenced by the service
> * resource name
> * declared media type and charset hints
> * optional declared content length and digest
> * a truncation flag
> Input must be bounded, and deadline or cancellation must terminate the associated parse work.
> The reply reuses the typed {{Document}} contract introduced by #2961 rather than defining a second parse-result model.
> The current direction discussed in #2961 is to place this on the experimental v2 surface so that it does not gate the 4.0.0 release.
> h4. Decided in the branch (2026-09-02)
> * bounded unary {{TikaV2.ParseBytes}} on the experimental v2 surface; the reply is the typed {{Document}} of #2961
> * provenance carried and echoed: correlation id (as {{Document.id}}, the caller's key), source/effective/base URI, resource name, truncation flag. Declared media type, charset, length and digest keep field numbers 7-10 reserved until they have verifiable semantics
> * payload bounded by a content cap ({{grpc.parseBytesMaxContentBytes}}, 64 MiB by default), separate from the gRPC inbound limit; bytes routed like a tika-server request body, inline up to {{pipes.maxInlineBytes}}, spooled above it
> * branch [TIKA-4795-parseBytes|https://github.com/ai-pipestream/tika/tree/TIKA-4795-parseBytes], stacked on #2961; PR once #2961 lands
> h4. Non-goals
> This issue does not define the structured content tree or its Markdown projection, the embedded-document representation, extension payloads such as {{google.protobuf.Any}}, or downstream NLP and embedding enrichment. Those remain separate TIKA-4766 stages and follow-up issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)