You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "dschmidt (via GitHub)" <gi...@apache.org> on 2026/08/19 16:06:07 UTC
[PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
dschmidt opened a new pull request, #3039:
URL: https://github.com/apache/tika/pull/3039
When parsing through tika-pipes (tika-server /unpack and /unpack/all, /async, /pipes), a caller can steer parser selection by supplying a filename, but supplying the correct `Content-Type` header alone does not work.
Example: NEF is TIFF-based with no content magic; `image/x-raw-nikon` is a `*.nef` glob plus `sub-class-of image/tiff`. By data alone a NEF detects as `image/tiff` (→ TiffParser); only the filename yields `image/x-raw-nikon`. A client streaming a NEF with `Content-Type: image/x-raw-nikon` but no filename still gets TiffParser.
Root cause: `PipesWorker.parseFromTuple` builds a fresh `Metadata` for fetch/detection (deliberately isolated from the caller's tuple metadata) and carries only `RESOURCE_NAME_KEY` across that boundary. The caller's `Content-Type`, which `TikaResource.fillMetadata` does set, is dropped before detection. Core detection would honor it: `MimeTypes.detect` applies a `Content-Type` hint via `applyHint`, keeping it when it equals or specializes the magic-detected type.
This change carries `HttpHeaders.CONTENT_TYPE` across the boundary alongside the resource name, as a soft hint.
**Security:** only the soft hint is carried, deliberately **not** `CONTENT_TYPE_USER_OVERRIDE`/`CONTENT_TYPE_PARSER_OVERRIDE`. `applyHint` constrains the hint to types that equal or specialize the magic-detected type, so a caller can refine within the hierarchy (`image/tiff` → `image/x-raw-nikon`) but cannot force an unrelated type. This grants no more routing power than the already-carried filename, and detection overwrites `HttpHeaders.CONTENT_TYPE` with the detected type before parsing, so no parser sees an unvalidated caller value. Carrying an override key would instead let a caller bypass detection unconditionally.
This softens the worker's deliberate metadata isolation for two fields; feedback welcome on whether the soft-hint scope is the right contract.
Unit test covers the carry: resource name and `Content-Type` carried, override key never carried, null/blank no-ops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #3039:
URL: https://github.com/apache/tika/pull/3039#issuecomment-5346813789
claude has some input. let me know what you think.
```
1. HIGH — the NEF example only works if PR #3037 merges first. At this
PR's head, image/x-raw-nikon (tika-mimetypes.xml:7234) has no
sub-class-of image/tiff — the PR description's claim that it does is
wrong for main today; #3037 is what adds it. Standalone, applyHint
rejects the hint and a NEF still detects as image/tiff, so the headline
example in CHANGES and the carryCallerHints javadoc silently does
nothing. Two reviewers found this independently; I verified both files.
Fix: merge-order note (or land the sub-class-of here), and an e2e test
of the real-mime-database refinement case — nothing currently catches
this.
2. MEDIUM — CHANGES endpoint list is materially incomplete. Every parse
endpoint routes through parseFromTuple: /tika, /meta, /rmeta and
variants (the PR's own testMeta flipping 404→422 proves /meta), plus
tika-grpc and embedded PipesForkParser users. Listing only /unpack,
/unpack/all, /async, /pipes misleads upgraders — Copilot's /unpack/all
fix was cosmetic; the real gap remains.
3. MEDIUM — undocumented 3.x→4.x downgrade in the migration doc. In 3.x
the header was a hard override (CONTENT_TYPE_USER_OVERRIDE); now it's
refine-only. A 3.x client forcing text/plain onto arbitrary bytes gets
silently different behavior. migrating-tika-server-4x.adoc:427-430 says
Content-Type "still influences detection" — true post-PR but hides the
override→hint change. One paragraph fixes it.
4. MEDIUM — no negative-path tests for the security boundary. Nothing
asserts a non-specializing or garbage Content-Type is ignored — the
exact property the javadoc calls security-critical. A future applyHint
change could turn the hint into an override with nothing failing. Also:
for magic-less bytes everything specializes application/octet-stream,
so the hint effectively wins there — "only refines within the
hierarchy" is overstated in CHANGES (equal to filename power, so
acceptable, but say it).
5. MEDIUM — CHANGES entry sits in the already-rc'd "Release 4.0.0 -
8/18/2026" section, which rc1 froze without it (main is now
4.0.1-SNAPSHOT). If rc1 passes, released 4.0.0 won't contain TIKA-4825
while CHANGES claims it does. Correct only if you sink rc1 and respin —
which is exactly what dschmidt is asking you in the PR thread.
6. LOW — pre-existing, newly exposed: the carried value reaches
MimeTypes.forName, which registers every distinct syntactically-valid
unknown type into an unbounded map (slow memory growth under
adversarial unique types). Pre-existing on direct server endpoints;
this PR adds the pipes tuple path. Real fix is a non-registering lookup
in MimeTypes.detect — separate JIRA, not this PR.
7. LOW — cleanups: fillMetadata still stamps CONTENT_TYPE_USER_OVERRIDE
(TikaResource.java:263,341) which the worker now deliberately discards
— dead and misleading, natural companion cleanup. testMeta's 422 body
is unasserted (old test asserted the body; one assertContains closes
it). The 4-line call-site comment in parseFromTuple restates the
javadoc — trim to one line per repo policy.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #3039:
URL: https://github.com/apache/tika/pull/3039#issuecomment-5346623458
I don't view this as breaking. It is a good catch, but I'm ok w 4.0.1.
Let me know if you disagree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison merged PR #3039:
URL: https://github.com/apache/tika/pull/3039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "dschmidt (via GitHub)" <gi...@apache.org>.
dschmidt commented on PR #3039:
URL: https://github.com/apache/tika/pull/3039#issuecomment-5346643305
Just wanted to bring it up, so you can decide because it's a slight behavioral change. Fine with me either way :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3039:
URL: https://github.com/apache/tika/pull/3039#discussion_r3814943057
##########
tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesWorker.java:
##########
@@ -516,7 +510,33 @@ protected ParseDataOrPipesResult parseFromTuple() throws TikaException, Interrup
}
}
-
+ /**
+ * Carries the caller-supplied detection hints from the tuple metadata across the
+ * fresh-metadata boundary into the metadata used for fetch and detection.
+ * <p>
+ * Only the resource name and the {@code Content-Type} soft hint are carried.
+ * {@code Content-Type} is applied by {@code MimeTypes.detect} via {@code applyHint},
+ * which keeps it only when it equals or specializes the magic-detected type (e.g.
+ * {@code image/tiff} -> {@code image/x-raw-nikon} for a NEF supplied without a
+ * filename). The {@code CONTENT_TYPE_USER_OVERRIDE} key is deliberately NOT carried:
+ * it short-circuits detection unconditionally and would let a caller force any type.
+ *
+ * @param tupleMetadata the caller-supplied metadata (may be null)
+ * @param target the fresh metadata used for fetch and detection
+ */
+ static void carryCallerHints(Metadata tupleMetadata, Metadata target) {
+ if (tupleMetadata == null) {
+ return;
+ }
+ String suppliedName = tupleMetadata.get(TikaCoreProperties.RESOURCE_NAME_KEY);
+ if (!StringUtils.isBlank(suppliedName)) {
+ target.set(TikaCoreProperties.RESOURCE_NAME_KEY, suppliedName);
+ }
+ String suppliedContentType = tupleMetadata.get(HttpHeaders.CONTENT_TYPE);
+ if (!StringUtils.isBlank(suppliedContentType)) {
+ target.set(HttpHeaders.CONTENT_TYPE, suppliedContentType);
+ }
Review Comment:
Consider normalizing the carried hints before setting them on `target`. As written, leading/trailing whitespace (or other non-canonical formatting) will be preserved, which can reduce the effectiveness of detection hints. A concrete improvement is to trim the values (e.g., use a trim-to-null approach) before `target.set(...)`; for `Content-Type`, optionally parse/validate and only carry it when it parses as a media type.
##########
tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/PipesWorkerCallerHintsTest.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.pipes.core.server;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.metadata.HttpHeaders;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+
+/**
+ * Unit tests for {@link PipesWorker#carryCallerHints(Metadata, Metadata)}, which carries the
+ * caller-supplied detection hints across the worker's fresh-metadata boundary.
+ */
+public class PipesWorkerCallerHintsTest {
+
+ @Test
+ public void testCarriesResourceNameAndContentType() {
+ Metadata tuple = new Metadata();
+ tuple.set(TikaCoreProperties.RESOURCE_NAME_KEY, "photo.nef");
+ tuple.set(HttpHeaders.CONTENT_TYPE, "image/x-raw-nikon");
+
+ Metadata target = new Metadata();
+ PipesWorker.carryCallerHints(tuple, target);
+
+ assertEquals("photo.nef", target.get(TikaCoreProperties.RESOURCE_NAME_KEY));
+ assertEquals("image/x-raw-nikon", target.get(HttpHeaders.CONTENT_TYPE));
+ }
+
+ /**
+ * The Content-Type is carried only as a soft hint. The unconditional override key
+ * must never be carried, or a caller could force any type past detection.
+ */
+ @Test
+ public void testDoesNotCarryUserOverride() {
+ Metadata tuple = new Metadata();
+ tuple.set(TikaCoreProperties.CONTENT_TYPE_USER_OVERRIDE, "image/x-raw-nikon");
+
+ Metadata target = new Metadata();
+ PipesWorker.carryCallerHints(tuple, target);
+
+ assertNull(target.get(TikaCoreProperties.CONTENT_TYPE_USER_OVERRIDE));
Review Comment:
The PR description mentions explicitly not carrying `CONTENT_TYPE_PARSER_OVERRIDE` as well as `CONTENT_TYPE_USER_OVERRIDE`, but the tests only assert the user-override key is not propagated. Add a regression assertion (or a separate test) that `TikaCoreProperties.CONTENT_TYPE_PARSER_OVERRIDE` is never carried, to lock in the intended security contract and prevent future regressions.
##########
CHANGES.txt:
##########
@@ -493,6 +493,13 @@ Release 4.0.0 - 8/18/2026
that exceed the limit return PAYLOAD_LIMIT_EXCEEDED instead of causing
heap exhaustion; crash messages are also size-capped (TIKA-4793).
+ * Pipes now carries the caller-supplied Content-Type across the worker's
+ fresh-metadata boundary as a soft detection hint, so /unpack, /async and
+ /pipes can route on a client Content-Type (e.g. image/x-raw-nikon for a NEF
+ sent without a filename), not only on the filename. The hint only refines
+ within the magic-detected type hierarchy; the CONTENT_TYPE_USER_OVERRIDE key
+ is deliberately not carried (TIKA-4825).
Review Comment:
The PR description calls out `/unpack/all` in addition to `/unpack`, but the CHANGES entry lists only `/unpack, /async and /pipes`. If `/unpack/all` is also affected by the same pipes/worker path, consider updating this entry to include it (or reword to cover all relevant endpoints consistently).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TIKA-4825] carry caller Content-Type across the pipes worker boundary as a detection hint [tika]
Posted by "dschmidt (via GitHub)" <gi...@apache.org>.
dschmidt commented on PR #3039:
URL: https://github.com/apache/tika/pull/3039#issuecomment-5346398676
@tballison @THausherr is 4.0.0 already released? This could be a somewhat breaking change that might be worth getting into the major release if you can still cut a rc2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org