You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by ta...@apache.org on 2026/08/17 10:28:27 UTC
(tika) branch TIKA-4808-pre-4.0.0-release-fixes created (now d8625f0a4b)
This is an automated email from the ASF dual-hosted git repository.
tballison pushed a change to branch TIKA-4808-pre-4.0.0-release-fixes
in repository https://gitbox.apache.org/repos/asf/tika.git
at d8625f0a4b TIKA-4808: config isolation, plugin boundary by scope, stream resource hardening
This branch includes the following new commits:
new 7c2ec48c8b TIKA-4808: take one, documentation, licenses and other small fixes
new d8625f0a4b TIKA-4808: config isolation, plugin boundary by scope, stream resource hardening
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(tika) 02/02: TIKA-4808: config isolation, plugin boundary by scope, stream resource hardening
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch TIKA-4808-pre-4.0.0-release-fixes
in repository https://gitbox.apache.org/repos/asf/tika.git
commit d8625f0a4b4ca7ed1b1695b846b12e3f3597a967
Author: tallison <ta...@apache.org>
AuthorDate: Mon Aug 17 06:27:48 2026 -0400
TIKA-4808: config isolation, plugin boundary by scope, stream resource hardening
---
CHANGES.txt | 8 ++-
docs/modules/ROOT/pages/advanced/spooling.adoc | 4 +-
.../pages/migration-to-4x/migrating-to-4x.adoc | 11 +++-
.../ROOT/pages/pipes/plugins/writing-a-plugin.adoc | 21 ++----
tika-core/src/main/java/org/apache/tika/Tika.java | 42 +++++++-----
.../org/apache/tika/config/ParseContextConfig.java | 27 ++++----
.../java/org/apache/tika/config/package-info.java | 2 +-
.../java/org/apache/tika/detect/package-info.java | 2 +-
.../java/org/apache/tika/digest/DigestHelper.java | 6 +-
.../org/apache/tika/exception/package-info.java | 2 +-
.../org/apache/tika/extractor/package-info.java | 2 +-
.../java/org/apache/tika/io/ByteArraySource.java | 6 +-
.../org/apache/tika/io/CachingInputStream.java | 5 ++
.../java/org/apache/tika/io/CachingSource.java | 29 +++++---
.../main/java/org/apache/tika/io/FileSource.java | 2 +-
.../main/java/org/apache/tika/io/StreamCache.java | 40 +++++++----
.../java/org/apache/tika/io/TikaInputSource.java | 2 +-
.../java/org/apache/tika/io/TikaInputStream.java | 66 +++++++++----------
.../main/java/org/apache/tika/io/package-info.java | 2 +-
.../java/org/apache/tika/metadata/ISO19115.java | 2 +-
.../java/org/apache/tika/metadata/Metadata.java | 4 +-
.../org/apache/tika/metadata/package-info.java | 2 +-
.../java/org/apache/tika/mime/package-info.java | 2 +-
.../main/java/org/apache/tika/package-info.java | 2 +-
.../java/org/apache/tika/parser/ParsingIntent.java | 25 +++----
.../java/org/apache/tika/parser/package-info.java | 2 +-
.../tika/sax/BasicContentHandlerFactory.java | 3 +-
.../java/org/apache/tika/sax/package-info.java | 2 +-
.../org/apache/tika/sax/xpath/package-info.java | 2 +-
.../java/org/apache/tika/utils/package-info.java | 2 +-
.../tika/metadata/metadata-migration-3x-4x.json | 2 +-
.../apache/tika/TikaDetectStreamOwnershipTest.java | 77 ++++++++++++++++++++++
.../org/apache/tika/io/TikaInputStreamTest.java | 61 +++++++++++++++--
.../apache/tika/metadata/metadata-key-fields.json | 2 +-
.../org/apache/tika/metadata/metadata-keys.json | 2 +-
.../org/apache/tika/metadata/migration-overlay.tsv | 2 +-
tika-parent/pom.xml | 2 +-
.../geoinfo/GeographicInformationParser.java | 2 +-
tika-parsers/tika-parsers-ml/tika-vlm/pom.xml | 7 ++
.../tika/parser/RecursiveParserWrapperTest.java | 7 +-
.../java/org/apache/tika/parser/pdf/PDFParser.java | 11 ++--
.../apache/tika/parser/pkg/CompressorParser.java | 14 ++--
.../detect/zip/DefaultZipContainerDetector.java | 2 +-
.../org/apache/tika/async/cli/TikaAsyncCLI.java | 4 +-
.../tika/pipes/core/extractor/UnpackExtractor.java | 5 +-
tika-pipes/tika-pipes-plugins/pom.xml | 41 ++++++++++++
.../tika-pipes-atlassian-jwt/pom.xml | 6 +-
.../fetcher/atlassianjwt/AtlassianJwtFetcher.java | 16 ++++-
.../tika-pipes-plugins/tika-pipes-az-blob/pom.xml | 9 ---
.../tika-pipes-plugins/tika-pipes-csv/pom.xml | 9 ---
.../tika-pipes-plugins/tika-pipes-es/pom.xml | 4 --
.../tika-pipes-file-system/pom.xml | 9 ---
.../tika-pipes-plugins/tika-pipes-gcs/pom.xml | 9 ---
.../tika-pipes-google-drive/pom.xml | 6 +-
.../fetcher/googledrive/GoogleDriveFetcher.java | 3 +-
.../tika-pipes-plugins/tika-pipes-http/pom.xml | 4 --
.../tika/pipes/fetcher/http/HttpFetcher.java | 12 +++-
.../fetcher/http/config/HttpFetcherConfig.java | 3 +-
.../tika-pipes-plugins/tika-pipes-jdbc/pom.xml | 9 ---
.../pipes/reporter/jdbc/JDBCPipesReporter.java | 2 +
.../tika-pipes-plugins/tika-pipes-json/pom.xml | 11 +---
.../tika-pipes-plugins/tika-pipes-kafka/pom.xml | 9 ---
.../tika-pipes-microsoft-graph/pom.xml | 4 --
.../tika-pipes-opensearch/pom.xml | 9 ---
.../tika-pipes-plugins/tika-pipes-s3/pom.xml | 9 ---
.../apache/tika/pipes/fetcher/s3/S3Fetcher.java | 2 +
.../tika-pipes-plugins/tika-pipes-solr/pom.xml | 9 ---
.../config/loader/AbstractSpiComponentLoader.java | 14 +++-
.../tika/serialization/ConfigDeserializer.java | 7 +-
.../apache/tika/config/loader/TikaLoaderTest.java | 15 +++++
.../tika/serialization/ConfigDeserializerTest.java | 38 +++++++++++
.../test-default-parser-unknown-exclusion.json | 9 +++
.../tika/server/core/resource/TikaResource.java | 12 ++--
.../server/core/writer/CSVMessageBodyWriter.java | 2 +-
74 files changed, 512 insertions(+), 307 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index ddf54b492c..46975e6b85 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -126,7 +126,11 @@ Release 4.0.0 - ???
TikaInputStream. Every third-party Parser, Detector or
EmbeddedDocumentExtractor implementation must be updated; callers can wrap
with TikaInputStream.get(...). The Tika facade (Tika.parse/parseToString)
- still accepts an InputStream and is unaffected
+ still accepts an InputStream and is unaffected. Tika.detect(InputStream, ...)
+ no longer resets the stream to its original position: detection now reads
+ ahead through a TikaInputStream, so on return the caller's stream must be
+ treated as consumed. It still does not close the caller's stream, and any
+ temporary file spooled during detection is deleted before it returns
(TIKA-4399, TIKA-4541, TIKA-4569).
* TikaConfig and the org.apache.tika.config XML-configuration API are
@@ -195,7 +199,7 @@ Release 4.0.0 - ???
AmazonTranscribeConfig, MagikaDetector/SiegfriedDetector configs)
(TIKA-4758).
- * TikaInputStream no longer buffers by default. A stream is consumed in
+ * TikaInputStream no longer caches by default. A stream is consumed in
passthrough mode unless enableRewind() is called at position 0;
rewind()/getFile()/getPath() after reading without enableRewind() throw
instead of silently spooling. Digesters call enableRewind() themselves
diff --git a/docs/modules/ROOT/pages/advanced/spooling.adoc b/docs/modules/ROOT/pages/advanced/spooling.adoc
index 630706eda4..04b60f95b8 100644
--- a/docs/modules/ROOT/pages/advanced/spooling.adoc
+++ b/docs/modules/ROOT/pages/advanced/spooling.adoc
@@ -64,7 +64,9 @@ File file = tis.getFile();
* **Initialized with `Path`**: The file is used directly for random access. No spooling needed.
* **Initialized with `byte[]`**: The bytes are kept in memory. Spooling only on demand.
* **Initialized with `InputStream`**: `getPath()` or `getFile()` spools the stream to a
- temporary file, which is automatically cleaned up when the stream is closed.
+ temporary file, which is automatically cleaned up when the stream is closed. Spooling
+ requires position 0 (or a prior `enableRewind()` at position 0); reading first and
+ spooling later throws.
==== Benefits of Decentralized Spooling
diff --git a/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc b/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
index 7b8583e1c6..86661acde1 100644
--- a/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
+++ b/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
@@ -56,12 +56,21 @@ try (TikaInputStream tis = TikaInputStream.get(myInputStream)) {
----
The `Tika` facade (`Tika#parse`, `Tika#parseToString`, `Tika#detect`) still accepts a plain
-`InputStream` and is unaffected — if you only use the facade, nothing changes.
+`InputStream`. One behavior change: `Tika#detect(InputStream, ...)` no longer resets the stream
+to its original position — treat the stream as consumed after the call. It still does not close
+the caller's stream.
Why: passing `TikaInputStream` explicitly makes the spooling and rewind contract visible in the
signature instead of leaving every implementation to wrap defensively. See
xref:advanced/spooling.adoc[spooling] for the stream contract.
+`TikaInputStream` itself dropped several members in 4.0: `isTikaInputStream`, `cast`,
+`getPath(int)`, `get(InputStreamFactory)` (both overloads), `hasInputStreamFactory`,
+`getInputStreamFactory`, and `setOpenContainer(Object, long)` — along with the
+`InputStreamFactory` class. `get(byte[])` and `getFromContainer` no longer declare
+`IOException`. `enableRewind()` now throws a checked `IOException` when bytes have
+already been read.
+
[#thin-launcher-zip]
== `tika-app` and `tika-server` distributions: jar -> zip
diff --git a/docs/modules/ROOT/pages/pipes/plugins/writing-a-plugin.adoc b/docs/modules/ROOT/pages/pipes/plugins/writing-a-plugin.adoc
index f17674d81f..3bc6f3299a 100644
--- a/docs/modules/ROOT/pages/pipes/plugins/writing-a-plugin.adoc
+++ b/docs/modules/ROOT/pages/pipes/plugins/writing-a-plugin.adoc
@@ -219,16 +219,10 @@ If that file is missing or empty, the plugin loads and contributes nothing.
=== Collecting runtime dependencies
-`maven-dependency-plugin` copies runtime dependencies to `target/lib`, minus the boundary
-artifacts:
-
-[source,xml]
-----
-<properties>
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
-</properties>
-----
+`maven-dependency-plugin` copies runtime dependencies to `target/lib`. There is no
+exclusion list to maintain: every boundary artifact is declared `provided`, and
+`includeScope=runtime` excludes `provided` by definition. A boundary artifact can
+only end up in `lib/` if a pom re-declares it at compile scope -- do not.
[source,xml]
----
@@ -243,8 +237,6 @@ artifacts:
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
@@ -276,7 +268,7 @@ A plugin's classloader prefers the plugin's own classes over the host's. So if y
`MyFetcherFactory` implements a `FetcherFactory` that is a *different* `Class` object from the
host's. The host then finds no extensions, or fails casting one.
-That is what the `plugin.excluded.artifactIds` property above prevents, and it is the usual cause
+That is what the `provided` scoping above prevents, and it is the usual cause
of a plugin that loads cleanly and then behaves as if it were not there — or of a
`NoClassDefFoundError` / `ClassCastException` naming a Tika type.
@@ -372,7 +364,8 @@ java -Dtika.plugin.dev.mode=true ...
* `plugin.properties` at the zip root, with `plugin.id` and `plugin.class`.
* `@Extension` on the factory, never on the fetcher or emitter.
* `META-INF/extensions.idx` non-empty in `target/classes` after compiling.
-* `tika-core`, `tika-pipes-api`, `tika-plugins-core`, `tika-serialization`, `pf4j` and the logging
+* `tika-core`, `tika-pipes-api`, `tika-plugins-core`, `tika-serialization`,
+ `tika-pipes-core`, `tika-pipes-iterator-commons`, `pf4j` and the logging
implementations `provided`, and absent from `lib/`.
* The zip — not an unpacked directory — dropped in a `plugin-roots` directory.
* `getName()` unique against every other loaded plugin.
diff --git a/tika-core/src/main/java/org/apache/tika/Tika.java b/tika-core/src/main/java/org/apache/tika/Tika.java
index 5195cf9108..5e5a4d913c 100644
--- a/tika-core/src/main/java/org/apache/tika/Tika.java
+++ b/tika-core/src/main/java/org/apache/tika/Tika.java
@@ -30,6 +30,7 @@ import org.apache.tika.detect.DefaultDetector;
import org.apache.tika.detect.Detector;
import org.apache.tika.exception.TikaException;
import org.apache.tika.exception.WriteLimitReachedException;
+import org.apache.tika.io.TemporaryResources;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.language.translate.DefaultTranslator;
import org.apache.tika.language.translate.Translator;
@@ -129,12 +130,14 @@ public class Tika {
* in which case only the given document metadata is used for type
* detection.
* <p>
- * If the document stream supports the
- * {@link InputStream#markSupported() mark feature}, then the stream is
- * marked and reset to the original position before this method returns.
- * Only a limited number of bytes are read from the stream.
+ * The given document stream is <em>not</em> closed by this method, and any
+ * temporary file spooled during detection is deleted before it returns.
* <p>
- * The given document stream is <em>not</em> closed by this method.
+ * The stream <em>is</em> consumed: detection reads ahead through a
+ * {@link TikaInputStream}, so on return the caller's stream must be treated as
+ * advanced, not rewound. Callers that need to read the document afterwards should
+ * pass a {@link TikaInputStream} they own (which they can
+ * {@link TikaInputStream#rewind() rewind}) or re-open the source.
* <p>
* Unlike in the {@link #parse(InputStream, Metadata)} method, the
* given document metadata is <em>not</em> modified by this method.
@@ -148,7 +151,10 @@ public class Tika {
if (stream == null) {
return detector.detect(null, metadata, new ParseContext()).toString();
}
- try (TikaInputStream tis = TikaInputStream.get(stream)) {
+ // Dispose only what detection spooled, never the caller's stream. Closing the
+ // TikaInputStream would also dispose a caller-supplied one (get() returns it as-is).
+ try (TemporaryResources tmp = new TemporaryResources()) {
+ TikaInputStream tis = TikaInputStream.get(stream, tmp, metadata);
return detector.detect(tis, metadata, new ParseContext()).toString();
}
}
@@ -158,12 +164,14 @@ public class Tika {
* based on the content of the given document stream and the name of the
* document.
* <p>
- * If the document stream supports the
- * {@link InputStream#markSupported() mark feature}, then the stream is
- * marked and reset to the original position before this method returns.
- * Only a limited number of bytes are read from the stream.
+ * The given document stream is <em>not</em> closed by this method, and any
+ * temporary file spooled during detection is deleted before it returns.
* <p>
- * The given document stream is <em>not</em> closed by this method.
+ * The stream <em>is</em> consumed: detection reads ahead through a
+ * {@link TikaInputStream}, so on return the caller's stream must be treated as
+ * advanced, not rewound. Callers that need to read the document afterwards should
+ * pass a {@link TikaInputStream} they own (which they can
+ * {@link TikaInputStream#rewind() rewind}) or re-open the source.
*
* @param stream the document stream
* @param name document name
@@ -182,12 +190,14 @@ public class Tika {
* Detects the media type of the given document. The type detection is
* based on the content of the given document stream.
* <p>
- * If the document stream supports the
- * {@link InputStream#markSupported() mark feature}, then the stream is
- * marked and reset to the original position before this method returns.
- * Only a limited number of bytes are read from the stream.
+ * The given document stream is <em>not</em> closed by this method, and any
+ * temporary file spooled during detection is deleted before it returns.
* <p>
- * The given document stream is <em>not</em> closed by this method.
+ * The stream <em>is</em> consumed: detection reads ahead through a
+ * {@link TikaInputStream}, so on return the caller's stream must be treated as
+ * advanced, not rewound. Callers that need to read the document afterwards should
+ * pass a {@link TikaInputStream} they own (which they can
+ * {@link TikaInputStream#rewind() rewind}) or re-open the source.
*
* @param stream the document stream
* @return detected media type
diff --git a/tika-core/src/main/java/org/apache/tika/config/ParseContextConfig.java b/tika-core/src/main/java/org/apache/tika/config/ParseContextConfig.java
index 8f1dc03371..a08e501585 100644
--- a/tika-core/src/main/java/org/apache/tika/config/ParseContextConfig.java
+++ b/tika-core/src/main/java/org/apache/tika/config/ParseContextConfig.java
@@ -69,14 +69,10 @@ public class ParseContextConfig {
/**
* Retrieves runtime configuration from ParseContext.
* <p>
- * This method first checks if the config is already resolved in ParseContext
- * (via {@code context.get(configClass)}). If found, it returns immediately without
- * re-deserializing. This is efficient for embedded documents where the config
- * was already deserialized for the parent document.
- * <p>
- * If not found, it checks jsonConfigs for the config key and deserializes
- * the JSON. The deserialized config is cached in resolvedConfigs and also
- * set in the main ParseContext for future lookups.
+ * Resolution is per config key: the resolved-config cache, then JSON config
+ * (deserialized and cached). A class-keyed entry set programmatically via
+ * {@code context.set(configClass, config)} is honored only when no JSON config
+ * exists for the key, since several components may share one config class.
* <p>
* This method performs defensive checking: if the ParseContext has JSON configuration
* for the requested key but the ConfigDeserializer is not available on the classpath,
@@ -98,16 +94,17 @@ public class ParseContextConfig {
return defaultConfig;
}
- // First check if config is already resolved in ParseContext
- // (may have been set by a previous call or by user code)
- T existingConfig = context.get(configClass);
- if (existingConfig != null) {
- return existingConfig;
+ // Per-key resolution first: a class-keyed entry may belong to a different
+ // component that happens to share this config class.
+ T resolved = context.getResolvedConfig(configKey);
+ if (resolved != null) {
+ return resolved;
}
- // Check for JSON config
if (!context.hasJsonConfig(configKey)) {
- return defaultConfig;
+ // no JSON for this key: honor a programmatic, class-keyed override
+ T existingConfig = context.get(configClass);
+ return existingConfig != null ? existingConfig : defaultConfig;
}
// JSON config exists for this key - ConfigDeserializer MUST be available
diff --git a/tika-core/src/main/java/org/apache/tika/config/package-info.java b/tika-core/src/main/java/org/apache/tika/config/package-info.java
index 77a0559f7f..61ee292ef7 100644
--- a/tika-core/src/main/java/org/apache/tika/config/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/config/package-info.java
@@ -18,5 +18,5 @@
/**
* Tika configuration tools.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.config;
diff --git a/tika-core/src/main/java/org/apache/tika/detect/package-info.java b/tika-core/src/main/java/org/apache/tika/detect/package-info.java
index dede49cfb6..a579b63597 100644
--- a/tika-core/src/main/java/org/apache/tika/detect/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/detect/package-info.java
@@ -18,5 +18,5 @@
/**
* Media type detection.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.detect;
diff --git a/tika-core/src/main/java/org/apache/tika/digest/DigestHelper.java b/tika-core/src/main/java/org/apache/tika/digest/DigestHelper.java
index 07bc9c4e6f..a5f751b1ae 100644
--- a/tika-core/src/main/java/org/apache/tika/digest/DigestHelper.java
+++ b/tika-core/src/main/java/org/apache/tika/digest/DigestHelper.java
@@ -80,10 +80,8 @@ public class DigestHelper {
Digester digester = digesterFactory.build();
- // Handle embedded stream translation if needed (e.g., for OLE2 objects in TikaInputStream's
- // open container). The translator consumes `tis` to produce the translated bytes that get
- // digested, so we enableRewind() before and rewind() after — otherwise the caller (typically
- // AutoDetectParser) would see an exhausted stream and fail its zero-byte probe (TIKA-XXXX).
+ // The translator consumes `tis` (e.g. OLE2), so enableRewind() before and rewind()
+ // after -- otherwise the caller would see an exhausted stream.
if (EMBEDDED_STREAM_TRANSLATOR.shouldTranslate(tis, metadata)) {
tis.enableRewind();
try (TemporaryResources tmp = new TemporaryResources()) {
diff --git a/tika-core/src/main/java/org/apache/tika/exception/package-info.java b/tika-core/src/main/java/org/apache/tika/exception/package-info.java
index 80ab125814..13c7b98c8d 100644
--- a/tika-core/src/main/java/org/apache/tika/exception/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/exception/package-info.java
@@ -18,5 +18,5 @@
/**
* Tika exception.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.exception;
diff --git a/tika-core/src/main/java/org/apache/tika/extractor/package-info.java b/tika-core/src/main/java/org/apache/tika/extractor/package-info.java
index 3d3e92b525..ddb1dfd33c 100644
--- a/tika-core/src/main/java/org/apache/tika/extractor/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/extractor/package-info.java
@@ -18,5 +18,5 @@
/**
* Extraction of component documents.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.extractor;
diff --git a/tika-core/src/main/java/org/apache/tika/io/ByteArraySource.java b/tika-core/src/main/java/org/apache/tika/io/ByteArraySource.java
index 6a2046cdf4..d2d3bbd61a 100644
--- a/tika-core/src/main/java/org/apache/tika/io/ByteArraySource.java
+++ b/tika-core/src/main/java/org/apache/tika/io/ByteArraySource.java
@@ -58,6 +58,10 @@ class ByteArraySource extends InputStream implements TikaInputSource {
@Override
public int read(byte[] b, int off, int len) {
+ // InputStream contract: a zero-length read returns 0, even at EOF
+ if (len == 0) {
+ return 0;
+ }
if (position >= length) {
return -1;
}
@@ -116,7 +120,7 @@ class ByteArraySource extends InputStream implements TikaInputSource {
}
@Override
- public void enableRewind() {
+ public void enableRewind() throws IOException {
// No-op: byte array is always rewindable
}
diff --git a/tika-core/src/main/java/org/apache/tika/io/CachingInputStream.java b/tika-core/src/main/java/org/apache/tika/io/CachingInputStream.java
index b9474b1cae..c17537ba22 100644
--- a/tika-core/src/main/java/org/apache/tika/io/CachingInputStream.java
+++ b/tika-core/src/main/java/org/apache/tika/io/CachingInputStream.java
@@ -182,6 +182,11 @@ class CachingInputStream extends InputStream {
cache.close();
}
+ /** For the owner to close after a spill. */
+ InputStream getSource() {
+ return source;
+ }
+
@Override
public int available() throws IOException {
// Return cached bytes available from current position
diff --git a/tika-core/src/main/java/org/apache/tika/io/CachingSource.java b/tika-core/src/main/java/org/apache/tika/io/CachingSource.java
index 7d4c648b99..944bd54c45 100644
--- a/tika-core/src/main/java/org/apache/tika/io/CachingSource.java
+++ b/tika-core/src/main/java/org/apache/tika/io/CachingSource.java
@@ -43,6 +43,8 @@ class CachingSource extends InputStream implements TikaInputSource {
private final TemporaryResources tmp;
private final Metadata metadata;
+ // temp-file suffix for threshold spills, which precede any getPath(suffix) call
+ private final String suffix;
private long length;
// Passthrough mode: just a BufferedInputStream
@@ -56,11 +58,16 @@ class CachingSource extends InputStream implements TikaInputSource {
private Path spilledPath;
private InputStream fileStream;
private long filePosition; // Track position in file mode
+ // Retained after a spill so close() can still close it; not closed at spill
+ // time because an archive stream may still be in use.
+ private InputStream spilledSource;
- CachingSource(InputStream source, TemporaryResources tmp, long length, Metadata metadata) {
+ CachingSource(InputStream source, TemporaryResources tmp, long length, Metadata metadata,
+ String suffix) {
this.tmp = tmp;
this.length = length;
this.metadata = metadata;
+ this.suffix = suffix;
// Start in passthrough mode
this.passthroughStream = source instanceof BufferedInputStream
? (BufferedInputStream) source
@@ -178,20 +185,20 @@ class CachingSource extends InputStream implements TikaInputSource {
}
@Override
- public void enableRewind() {
+ public void enableRewind() throws IOException {
// Already in caching or file mode - no-op
if (cachingStream != null || fileStream != null) {
return;
}
if (passthroughPosition != 0) {
- throw new IllegalStateException(
+ throw new IOException(
"Cannot enable rewind: position is " + passthroughPosition +
", must be 0. Call enableRewind() before reading.");
}
// Switch to caching mode
- StreamCache cache = new StreamCache(tmp);
+ StreamCache cache = new StreamCache(tmp, suffix);
cachingStream = new CachingInputStream(passthroughStream, cache);
passthroughStream = null;
}
@@ -246,7 +253,8 @@ class CachingSource extends InputStream implements TikaInputSource {
// Get current position before closing cache
long currentPosition = cachingStream.getPosition();
- // Close only the cache, not the source stream (for archive support)
+ // close only the cache; close() releases the source later
+ spilledSource = cachingStream.getSource();
cachingStream.closeCacheOnly();
// Open file stream at current position
@@ -256,11 +264,9 @@ class CachingSource extends InputStream implements TikaInputSource {
}
filePosition = currentPosition;
- // Update length from file size
- long fileSize = Files.size(spilledPath);
- if (length == -1 || fileSize > 0) {
- length = fileSize;
- }
+ // The spooled size is ground truth, even when it is 0 and a
+ // Content-Length claimed otherwise
+ length = Files.size(spilledPath);
// Update metadata if not already set
if (metadata != null &&
@@ -289,5 +295,8 @@ class CachingSource extends InputStream implements TikaInputSource {
if (passthroughStream != null) {
passthroughStream.close();
}
+ if (spilledSource != null) {
+ spilledSource.close();
+ }
}
}
diff --git a/tika-core/src/main/java/org/apache/tika/io/FileSource.java b/tika-core/src/main/java/org/apache/tika/io/FileSource.java
index 79163f0ab2..143bb2d562 100644
--- a/tika-core/src/main/java/org/apache/tika/io/FileSource.java
+++ b/tika-core/src/main/java/org/apache/tika/io/FileSource.java
@@ -120,7 +120,7 @@ class FileSource extends InputStream implements TikaInputSource {
}
@Override
- public void enableRewind() {
+ public void enableRewind() throws IOException {
// No-op: file is always rewindable
}
diff --git a/tika-core/src/main/java/org/apache/tika/io/StreamCache.java b/tika-core/src/main/java/org/apache/tika/io/StreamCache.java
index d474b189aa..87a71fe931 100644
--- a/tika-core/src/main/java/org/apache/tika/io/StreamCache.java
+++ b/tika-core/src/main/java/org/apache/tika/io/StreamCache.java
@@ -41,18 +41,27 @@ class StreamCache implements Closeable {
private int memorySize;
// File storage (null until spill)
+ private String suffix;
private Path spillFile;
+ // one long-lived read handle; opening per read cost ~37x on byte-at-a-time readers
+ private RandomAccessFile reader;
private OutputStream spillOutputStream;
private long totalSize;
private boolean closed;
StreamCache(TemporaryResources tmp) {
- this(tmp, DEFAULT_MEMORY_THRESHOLD);
+ this(tmp, null, DEFAULT_MEMORY_THRESHOLD);
+ }
+
+ /** Suffix up front: a threshold spill precedes any getPath(suffix) call (TIKA-3903). */
+ StreamCache(TemporaryResources tmp, String suffix) {
+ this(tmp, suffix, DEFAULT_MEMORY_THRESHOLD);
}
- StreamCache(TemporaryResources tmp, int memoryThreshold) {
+ StreamCache(TemporaryResources tmp, String suffix, int memoryThreshold) {
this.tmp = tmp;
+ this.suffix = suffix;
this.memoryThreshold = memoryThreshold;
this.memoryBuffer = new byte[Math.min(memoryThreshold, 8192)];
this.memorySize = 0;
@@ -116,8 +125,6 @@ class StreamCache implements Closeable {
memoryBuffer = newBuffer;
}
- private String suffix;
-
private void spillToFile() throws IOException {
if (spillFile != null) {
return; // Already spilled
@@ -148,10 +155,9 @@ class StreamCache implements Closeable {
return memoryBuffer[(int) position] & 0xFF;
} else {
flushSpillStream();
- try (RandomAccessFile raf = new RandomAccessFile(spillFile.toFile(), "r")) {
- raf.seek(position);
- return raf.read();
- }
+ RandomAccessFile raf = reader();
+ raf.seek(position);
+ return raf.read();
}
}
@@ -170,10 +176,9 @@ class StreamCache implements Closeable {
return available;
} else {
flushSpillStream();
- try (RandomAccessFile raf = new RandomAccessFile(spillFile.toFile(), "r")) {
- raf.seek(position);
- return raf.read(b, off, available);
- }
+ RandomAccessFile raf = reader();
+ raf.seek(position);
+ return raf.read(b, off, available);
}
}
@@ -184,6 +189,13 @@ class StreamCache implements Closeable {
return new CacheInputStream(offset);
}
+ private RandomAccessFile reader() throws IOException {
+ if (reader == null) {
+ reader = new RandomAccessFile(spillFile.toFile(), "r");
+ }
+ return reader;
+ }
+
private void flushSpillStream() throws IOException {
if (spillOutputStream != null) {
spillOutputStream.flush();
@@ -242,6 +254,10 @@ class StreamCache implements Closeable {
spillOutputStream.close();
spillOutputStream = null;
}
+ if (reader != null) {
+ reader.close();
+ reader = null;
+ }
// spillFile cleanup is handled by TemporaryResources
}
diff --git a/tika-core/src/main/java/org/apache/tika/io/TikaInputSource.java b/tika-core/src/main/java/org/apache/tika/io/TikaInputSource.java
index 1620614f99..4c6c6e88df 100644
--- a/tika-core/src/main/java/org/apache/tika/io/TikaInputSource.java
+++ b/tika-core/src/main/java/org/apache/tika/io/TikaInputSource.java
@@ -64,5 +64,5 @@ interface TikaInputSource extends Closeable {
*
* @throws IllegalStateException if position is not 0
*/
- void enableRewind();
+ void enableRewind() throws IOException;
}
diff --git a/tika-core/src/main/java/org/apache/tika/io/TikaInputStream.java b/tika-core/src/main/java/org/apache/tika/io/TikaInputStream.java
index 94b6db04b1..6eb1cf90d7 100644
--- a/tika-core/src/main/java/org/apache/tika/io/TikaInputStream.java
+++ b/tika-core/src/main/java/org/apache/tika/io/TikaInputStream.java
@@ -47,7 +47,8 @@ import org.apache.tika.utils.StringUtils;
* <ul>
* <li>{@link ByteArraySource} for byte[] inputs - no caching needed</li>
* <li>{@link FileSource} for Path/File inputs - direct file access</li>
- * <li>{@link CachingSource} for InputStream inputs - caches bytes as read</li>
+ * <li>{@link CachingSource} for InputStream inputs - passthrough by default;
+ * caches bytes only after {@link #enableRewind()}</li>
* </ul>
*
* @since Apache Tika 0.8
@@ -57,7 +58,7 @@ public class TikaInputStream extends TaggedInputStream {
private static final int MAX_CONSECUTIVE_EOFS = 1000;
private static final int BLOB_SIZE_THRESHOLD = 1024 * 1024;
- protected TemporaryResources tmp;
+ private final TemporaryResources tmp;
private long position = 0;
private long mark = -1;
@@ -69,14 +70,6 @@ public class TikaInputStream extends TaggedInputStream {
// ========== Constructors ==========
- /**
- * Protected constructor for subclasses.
- */
- protected TikaInputStream(InputStream stream, long length) {
- super(stream);
- this.tmp = null;
- }
-
/**
* Strategy-based constructor.
* TikaInputSource extends InputStream, so we pass it directly to super().
@@ -104,7 +97,7 @@ public class TikaInputStream extends TaggedInputStream {
return (TikaInputStream) stream;
}
String ext = getExtension(metadata);
- TikaInputSource inputSource = new CachingSource(stream, tmp, -1, metadata);
+ TikaInputSource inputSource = new CachingSource(stream, tmp, -1, metadata, ext);
return new TikaInputStream(inputSource, tmp, ext);
}
@@ -116,11 +109,11 @@ public class TikaInputStream extends TaggedInputStream {
return get(stream, new TemporaryResources(), metadata);
}
- public static TikaInputStream get(byte[] data) throws IOException {
+ public static TikaInputStream get(byte[] data) {
return get(data, new Metadata());
}
- public static TikaInputStream get(byte[] data, Metadata metadata) throws IOException {
+ public static TikaInputStream get(byte[] data, Metadata metadata) {
metadata.set(HttpHeaders.CONTENT_LENGTH, Integer.toString(data.length));
String ext = getExtension(metadata);
TemporaryResources tmp = new TemporaryResources();
@@ -181,7 +174,7 @@ public class TikaInputStream extends TaggedInputStream {
String ext = getExtension(metadata);
TemporaryResources tmp = new TemporaryResources();
TikaInputSource inputSource = new CachingSource(
- new BufferedInputStream(blob.getBinaryStream()), tmp, length, metadata);
+ new BufferedInputStream(blob.getBinaryStream()), tmp, length, metadata, ext);
return new TikaInputStream(inputSource, tmp, ext);
}
}
@@ -242,12 +235,11 @@ public class TikaInputStream extends TaggedInputStream {
String ext = getExtension(metadata);
TemporaryResources tmp = new TemporaryResources();
TikaInputSource inputSource = new CachingSource(
- new BufferedInputStream(connection.getInputStream()), tmp, length, metadata);
+ new BufferedInputStream(connection.getInputStream()), tmp, length, metadata, ext);
return new TikaInputStream(inputSource, tmp, ext);
}
- public static TikaInputStream getFromContainer(Object openContainer, long length, Metadata metadata)
- throws IOException {
+ public static TikaInputStream getFromContainer(Object openContainer, long length, Metadata metadata) {
TikaInputStream tis = TikaInputStream.get(new byte[0], metadata);
tis.setOpenContainer(openContainer);
tis.setLength(length);
@@ -340,18 +332,20 @@ public class TikaInputStream extends TaggedInputStream {
public int peek(byte[] buffer) throws IOException {
int n = 0;
mark(buffer.length);
-
- int m = read(buffer);
- while (m != -1) {
- n += m;
- if (n < buffer.length) {
- m = read(buffer, n, buffer.length - n);
- } else {
- m = -1;
+ // reset in finally: a throw mid-read must not leave the stream advanced
+ try {
+ int m = read(buffer);
+ while (m != -1) {
+ n += m;
+ if (n < buffer.length) {
+ m = read(buffer, n, buffer.length - n);
+ } else {
+ m = -1;
+ }
}
+ } finally {
+ reset();
}
-
- reset();
return n;
}
@@ -401,6 +395,10 @@ public class TikaInputStream extends TaggedInputStream {
return source != null && source.getLength() != -1;
}
+ /**
+ * The stream length. For a stream-backed instance with no declared length this
+ * spools the entire remaining stream to a temporary file to measure it.
+ */
public long getLength() throws IOException {
if (overrideLength >= 0) {
return overrideLength;
@@ -422,10 +420,6 @@ public class TikaInputStream extends TaggedInputStream {
return position;
}
- protected void setPosition(long position) {
- this.position = position;
- }
-
private void setLength(long length) {
this.overrideLength = length;
}
@@ -435,7 +429,10 @@ public class TikaInputStream extends TaggedInputStream {
}
public void removeCloseShield() {
- this.closeShieldDepth--;
+ // floored: an unmatched remove must not cancel a later caller's shield
+ if (closeShieldDepth > 0) {
+ closeShieldDepth--;
+ }
}
public boolean isCloseShield() {
@@ -477,9 +474,10 @@ public class TikaInputStream extends TaggedInputStream {
* For streaming-only operations (e.g., HTML parsing), skip this call
* to avoid unnecessary caching overhead.
*
- * @throws IllegalStateException if position is not 0
+ * @throws IOException if bytes have already been read from the stream
+ * (position is not 0); rewind support cannot be enabled retroactively
*/
- public void enableRewind() {
+ public void enableRewind() throws IOException {
TikaInputSource source = inputSource();
if (source != null) {
source.enableRewind();
diff --git a/tika-core/src/main/java/org/apache/tika/io/package-info.java b/tika-core/src/main/java/org/apache/tika/io/package-info.java
index 36c7274da5..3c828b965c 100644
--- a/tika-core/src/main/java/org/apache/tika/io/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/io/package-info.java
@@ -18,5 +18,5 @@
/**
* IO utilities.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.io;
diff --git a/tika-core/src/main/java/org/apache/tika/metadata/ISO19115.java b/tika-core/src/main/java/org/apache/tika/metadata/ISO19115.java
index 870e5445da..c93f0e6151 100644
--- a/tika-core/src/main/java/org/apache/tika/metadata/ISO19115.java
+++ b/tika-core/src/main/java/org/apache/tika/metadata/ISO19115.java
@@ -66,7 +66,7 @@ public interface ISO19115 {
Property OTHER_CONSTRAINTS = Property.externalTextBag("iso19115:other-constraints");
- Property USER_CONSTRAINTS = Property.externalTextBag("iso19115:user-constraints");
+ Property USE_CONSTRAINTS = Property.externalTextBag("iso19115:use-constraints");
Property GEOGRAPHIC_IDENTIFIER_CODE = Property.externalTextBag("iso19115:geographic-identifier-code");
diff --git a/tika-core/src/main/java/org/apache/tika/metadata/Metadata.java b/tika-core/src/main/java/org/apache/tika/metadata/Metadata.java
index b76bbe306e..d5a9467dbf 100644
--- a/tika-core/src/main/java/org/apache/tika/metadata/Metadata.java
+++ b/tika-core/src/main/java/org/apache/tika/metadata/Metadata.java
@@ -101,13 +101,13 @@ public class Metadata implements Serializable {
* this is skipped (with a WARN). Configurable policy belongs in a
* {@link org.apache.tika.metadata.writelimiter.StandardMetadataLimiter}.
*/
- public static final int MAX_PREFIX_ROUTE_NAME_LENGTH = 1024;
+ static final int MAX_PREFIX_ROUTE_NAME_LENGTH = 1024;
/**
* Safety net on {@link #add(KeyPrefix, String, String)}: after this many distinct
* prefix-route names in one Metadata instance, further new names are skipped (WARN once).
*/
- public static final int MAX_PREFIX_ROUTE_NAMES = 10_000;
+ static final int MAX_PREFIX_ROUTE_NAMES = 10_000;
/**
* Some parsers will have the date as a ISO-8601 string
* already, and will set that into the Metadata object.
diff --git a/tika-core/src/main/java/org/apache/tika/metadata/package-info.java b/tika-core/src/main/java/org/apache/tika/metadata/package-info.java
index 02fcae3ec8..b85b43ce0d 100644
--- a/tika-core/src/main/java/org/apache/tika/metadata/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/metadata/package-info.java
@@ -18,5 +18,5 @@
/**
* Multi-valued metadata container, and set of constant metadata fields.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.metadata;
diff --git a/tika-core/src/main/java/org/apache/tika/mime/package-info.java b/tika-core/src/main/java/org/apache/tika/mime/package-info.java
index 104dc3acf9..aa45c694e2 100644
--- a/tika-core/src/main/java/org/apache/tika/mime/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/mime/package-info.java
@@ -18,5 +18,5 @@
/**
* Media type information.
*/
-@aQute.bnd.annotation.Version("1.2.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.mime;
diff --git a/tika-core/src/main/java/org/apache/tika/package-info.java b/tika-core/src/main/java/org/apache/tika/package-info.java
index cf4352ddcb..5a189641a6 100644
--- a/tika-core/src/main/java/org/apache/tika/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/package-info.java
@@ -18,5 +18,5 @@
/**
* Apache Tika.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika;
diff --git a/tika-core/src/main/java/org/apache/tika/parser/ParsingIntent.java b/tika-core/src/main/java/org/apache/tika/parser/ParsingIntent.java
index 60f11101d9..bdc995afe5 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/ParsingIntent.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/ParsingIntent.java
@@ -17,24 +17,17 @@
package org.apache.tika.parser;
/**
- * Marker class to indicate parsing intent in ParseContext.
+ * Parsing intent, set in the ParseContext before detection.
* <p>
- * When set in the ParseContext before detection, this signals to detectors
- * that parsing will follow detection. Detectors can use this hint to perform
- * additional preparation (like salvaging corrupted ZIP files) that would
- * benefit the subsequent parse operation.
+ * {@link #WILL_PARSE} signals detectors that parsing will follow, so they may
+ * prepare durable state for the parser (e.g. salvage a corrupted ZIP into the
+ * open container). Set automatically by {@link AutoDetectParser}.
* <p>
- * This is automatically set by {@link AutoDetectParser} before calling
- * the detector.
+ * Consumers must compare against a specific constant, never test for mere
+ * presence, so that new intents can be added without changing their behavior.
*/
-public final class ParsingIntent {
+public enum ParsingIntent {
- /**
- * Singleton instance indicating that parsing will follow detection.
- */
- public static final ParsingIntent WILL_PARSE = new ParsingIntent();
-
- private ParsingIntent() {
- // Private constructor for singleton
- }
+ /** Parsing will follow detection. */
+ WILL_PARSE
}
diff --git a/tika-core/src/main/java/org/apache/tika/parser/package-info.java b/tika-core/src/main/java/org/apache/tika/parser/package-info.java
index 10df69e1e6..8142132eeb 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/package-info.java
@@ -18,5 +18,5 @@
/**
* Tika parsers.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.parser;
diff --git a/tika-core/src/main/java/org/apache/tika/sax/BasicContentHandlerFactory.java b/tika-core/src/main/java/org/apache/tika/sax/BasicContentHandlerFactory.java
index 09d7730a8d..6e787625a4 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/BasicContentHandlerFactory.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/BasicContentHandlerFactory.java
@@ -118,8 +118,9 @@ public class BasicContentHandlerFactory implements StreamingContentHandlerFactor
}
/** Accepted spellings, for error messages. */
+ // String.join defeats compile-time constant inlining into downstream jars
public static final String VALID_HANDLER_TYPE_NAMES =
- "text, txt, html, xml, body, markdown, md, ignore";
+ String.join(", ", "text", "txt", "html", "xml", "body", "markdown", "md", "ignore");
/**
* Parses a string into a handler type.
diff --git a/tika-core/src/main/java/org/apache/tika/sax/package-info.java b/tika-core/src/main/java/org/apache/tika/sax/package-info.java
index 3c0b4ba48d..f2522c4e1f 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/package-info.java
@@ -18,5 +18,5 @@
/**
* SAX utilities.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.sax;
diff --git a/tika-core/src/main/java/org/apache/tika/sax/xpath/package-info.java b/tika-core/src/main/java/org/apache/tika/sax/xpath/package-info.java
index f9c1801bd6..1a12e29aa8 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/xpath/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/xpath/package-info.java
@@ -18,5 +18,5 @@
/**
* XPath utilities
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.sax.xpath;
diff --git a/tika-core/src/main/java/org/apache/tika/utils/package-info.java b/tika-core/src/main/java/org/apache/tika/utils/package-info.java
index 04ea52e5cf..be106f7415 100644
--- a/tika-core/src/main/java/org/apache/tika/utils/package-info.java
+++ b/tika-core/src/main/java/org/apache/tika/utils/package-info.java
@@ -18,5 +18,5 @@
/**
* Utilities.
*/
-@aQute.bnd.annotation.Version("1.0.0")
+@aQute.bnd.annotation.Version("2.0.0")
package org.apache.tika.utils;
diff --git a/tika-core/src/main/resources/org/apache/tika/metadata/metadata-migration-3x-4x.json b/tika-core/src/main/resources/org/apache/tika/metadata/metadata-migration-3x-4x.json
index dfbf078146..60fe856431 100644
--- a/tika-core/src/main/resources/org/apache/tika/metadata/metadata-migration-3x-4x.json
+++ b/tika-core/src/main/resources/org/apache/tika/metadata/metadata-migration-3x-4x.json
@@ -71,7 +71,7 @@
{"v3":"TransferOptionsOnlineProtocol ","v4":"iso19115:transfer-options-online-protocol"},
{"v3":"Upper Left","v4":"gdal:upper-left"},
{"v3":"Upper Right","v4":"gdal:upper-right"},
-{"v3":"UserConstraints ","v4":"iso19115:user-constraints"},
+{"v3":"UserConstraints ","v4":"iso19115:use-constraints"},
{"v3":"X-TIKA:EXCEPTION:container_exception","v4":"tk:exception:container-exception"},
{"v3":"X-TIKA:EXCEPTION:embedded_bytes_exception","v4":"tk:exception:embedded-bytes-exception"},
{"v3":"X-TIKA:EXCEPTION:embedded_exception","v4":"tk:exception:embedded-exception"},
diff --git a/tika-core/src/test/java/org/apache/tika/TikaDetectStreamOwnershipTest.java b/tika-core/src/test/java/org/apache/tika/TikaDetectStreamOwnershipTest.java
new file mode 100644
index 0000000000..0b1cb319dd
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/TikaDetectStreamOwnershipTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+
+/** {@link Tika#detect} peeks: it must dispose only what it spooled itself. */
+public class TikaDetectStreamOwnershipTest {
+
+ private static final byte[] DATA = "hello world\n".getBytes(StandardCharsets.UTF_8);
+
+ private static class CloseCountingInputStream extends ByteArrayInputStream {
+ int closes = 0;
+
+ CloseCountingInputStream(byte[] buf) {
+ super(buf);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closes++;
+ super.close();
+ }
+ }
+
+ @Test
+ public void testDetectDoesNotCloseCallersStream() throws Exception {
+ Tika tika = new Tika();
+ CloseCountingInputStream stream = new CloseCountingInputStream(DATA);
+ tika.detect(stream, new Metadata());
+ assertEquals(0, stream.closes,
+ "detect() must not close a stream it was handed; the javadoc promises this");
+ }
+
+ /** get() returns a caller-supplied TikaInputStream as-is, so disposing it here
+ * would delete the caller's temp file. */
+ @Test
+ public void testDetectDoesNotDisposeCallerOwnedTikaInputStream() throws Exception {
+ Tika tika = new Tika();
+ try (TikaInputStream tis = TikaInputStream.get(DATA)) {
+ Path spooled = tis.getPath();
+ assertTrue(Files.exists(spooled), "precondition: caller spooled a temp file");
+
+ tika.detect(tis, new Metadata());
+
+ assertTrue(Files.exists(spooled),
+ "detect() must not dispose a caller-owned TikaInputStream's temp file");
+ assertTrue(tis.getPath().toFile().exists(), "the stream must still be usable");
+ }
+ }
+}
diff --git a/tika-core/src/test/java/org/apache/tika/io/TikaInputStreamTest.java b/tika-core/src/test/java/org/apache/tika/io/TikaInputStreamTest.java
index 2f8915b5f1..ac86ec3b05 100644
--- a/tika-core/src/test/java/org/apache/tika/io/TikaInputStreamTest.java
+++ b/tika-core/src/test/java/org/apache/tika/io/TikaInputStreamTest.java
@@ -684,7 +684,7 @@ public class TikaInputStreamTest {
try (TemporaryResources tmp = new TemporaryResources()) {
CachingSource source = new CachingSource(
- new ByteArrayInputStream(data), tmp, -1, metadata);
+ new ByteArrayInputStream(data), tmp, -1, metadata, null);
source.enableRewind(); // Enable caching for spill support
// Read all data
@@ -720,7 +720,7 @@ public class TikaInputStreamTest {
try (TemporaryResources tmp = new TemporaryResources()) {
CachingSource source = new CachingSource(
- new ByteArrayInputStream(data), tmp, -1, metadata);
+ new ByteArrayInputStream(data), tmp, -1, metadata, null);
source.enableRewind(); // Enable caching for seek/spill support
// Read and spill
@@ -741,7 +741,7 @@ public class TikaInputStreamTest {
try (TemporaryResources tmp = new TemporaryResources()) {
CachingSource source = new CachingSource(
- new ByteArrayInputStream(data), tmp, -1, null);
+ new ByteArrayInputStream(data), tmp, -1, null, null);
source.enableRewind(); // Enable caching for seek support
// Read first 5 bytes
@@ -767,7 +767,7 @@ public class TikaInputStreamTest {
try (TemporaryResources tmp = new TemporaryResources()) {
CachingSource source = new CachingSource(
- new ByteArrayInputStream(data), tmp, -1, null);
+ new ByteArrayInputStream(data), tmp, -1, null, null);
source.enableRewind(); // Enable caching for spill/seek support
// Read first 5 bytes
@@ -864,7 +864,7 @@ public class TikaInputStreamTest {
tis.read(); // Read one byte, position is now 1
assertEquals(1, tis.getPosition());
- assertThrows(IllegalStateException.class, tis::enableRewind,
+ assertThrows(IOException.class, tis::enableRewind,
"enableRewind() should throw when position != 0");
}
}
@@ -996,4 +996,55 @@ public class TikaInputStreamTest {
private static String str(byte[] b) {
return new String(b, UTF_8);
}
+
+ private static class CloseCountingInputStream extends ByteArrayInputStream {
+ int closes = 0;
+
+ CloseCountingInputStream(byte[] buf) {
+ super(buf);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closes++;
+ super.close();
+ }
+ }
+
+ /**
+ * Spill must not close the source (an archive stream may still be in use),
+ * but close() must reach it exactly once.
+ */
+ @Test
+ public void testSourceClosedOnceAfterSpill() throws Exception {
+ CloseCountingInputStream source = new CloseCountingInputStream(new byte[64]);
+ Path spilled;
+ try (TikaInputStream tis = TikaInputStream.get(source)) {
+ spilled = tis.getPath();
+ assertEquals(0, source.closes, "source must stay open at spill time");
+ }
+ assertEquals(1, source.closes, "close() must close the spilled source exactly once");
+ assertFalse(Files.exists(spilled), "temp file must be deleted on close");
+ }
+
+ /** TIKA-3903: the suffix must survive a spill triggered by the memory threshold. */
+ @Test
+ public void testSuffixSurvivesThresholdSpill() throws Exception {
+ Metadata metadata = new Metadata();
+ metadata.set(TikaCoreProperties.RESOURCE_NAME_KEY, "scan.png");
+ // one byte past the StreamCache memory threshold
+ byte[] data = new byte[1024 * 1024 + 1];
+ try (TikaInputStream tis =
+ TikaInputStream.get(new ByteArrayInputStream(data), metadata)) {
+ tis.enableRewind();
+ // drain past the threshold so the cache spills before getPath()
+ byte[] buffer = new byte[8192];
+ while (tis.read(buffer) != -1) {
+ }
+ tis.rewind();
+ assertTrue(tis.getPath().getFileName().toString().endsWith(".png"),
+ "threshold spill must keep the original suffix, got: "
+ + tis.getPath().getFileName());
+ }
+ }
}
diff --git a/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-key-fields.json b/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-key-fields.json
index 7c79e191e7..3392c0e3be 100644
--- a/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-key-fields.json
+++ b/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-key-fields.json
@@ -231,7 +231,7 @@
{"class":"org.apache.tika.metadata.ISO19115","field":"TRANSFER_OPTIONS_ONLINE_NAME","key":"iso19115:transfer-options-online-name"},
{"class":"org.apache.tika.metadata.ISO19115","field":"TRANSFER_OPTIONS_ONLINE_PROFILE","key":"iso19115:transfer-options-online-profile"},
{"class":"org.apache.tika.metadata.ISO19115","field":"TRANSFER_OPTIONS_ONLINE_PROTOCOL","key":"iso19115:transfer-options-online-protocol"},
- {"class":"org.apache.tika.metadata.ISO19115","field":"USER_CONSTRAINTS","key":"iso19115:user-constraints"},
+ {"class":"org.apache.tika.metadata.ISO19115","field":"USE_CONSTRAINTS","key":"iso19115:use-constraints"},
{"class":"org.apache.tika.metadata.MAPI","field":"ATTACH_CONTENT_ID","key":"mapi:attach:content-id"},
{"class":"org.apache.tika.metadata.MAPI","field":"ATTACH_CONTENT_LOCATION","key":"mapi:attach:content-location"},
{"class":"org.apache.tika.metadata.MAPI","field":"ATTACH_DISPLAY_NAME","key":"mapi:attach:display-name"},
diff --git a/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-keys.json b/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-keys.json
index b0a7aea762..817b002655 100644
--- a/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-keys.json
+++ b/tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-keys.json
@@ -243,7 +243,7 @@
{"key":"iso19115:transfer-options-online-name","namespace":"iso19115","valueType":"TEXT","cardinality":"BAG","module":"tika-core"},
{"key":"iso19115:transfer-options-online-profile","namespace":"iso19115","valueType":"TEXT","cardinality":"BAG","module":"tika-core"},
{"key":"iso19115:transfer-options-online-protocol","namespace":"iso19115","valueType":"TEXT","cardinality":"BAG","module":"tika-core"},
- {"key":"iso19115:user-constraints","namespace":"iso19115","valueType":"TEXT","cardinality":"BAG","module":"tika-core"},
+ {"key":"iso19115:use-constraints","namespace":"iso19115","valueType":"TEXT","cardinality":"BAG","module":"tika-core"},
{"key":"iwork:build-version-history","namespace":"iwork","valueType":"TEXT","cardinality":"BAG","module":"tika-parser-apple-module"},
{"key":"iwork:document-id","namespace":"iwork","valueType":"TEXT","cardinality":"SIMPLE","module":"tika-parser-apple-module"},
{"key":"keynote:slides-height","namespace":"keynote","valueType":"TEXT","cardinality":"SIMPLE","module":"tika-parser-apple-module"},
diff --git a/tika-metadata-schema/src/test/resources/org/apache/tika/metadata/migration-overlay.tsv b/tika-metadata-schema/src/test/resources/org/apache/tika/metadata/migration-overlay.tsv
index fbd2496652..8288dbedaa 100644
--- a/tika-metadata-schema/src/test/resources/org/apache/tika/metadata/migration-overlay.tsv
+++ b/tika-metadata-schema/src/test/resources/org/apache/tika/metadata/migration-overlay.tsv
@@ -197,7 +197,7 @@ IdentificationInfoTopicCategory--> iso19115:identification-info-topic-category
ThesaurusNameDate iso19115:thesaurus-name-date
AccessContraints iso19115:access-constraints
OtherConstraints iso19115:other-constraints
-UserConstraints iso19115:user-constraints
+UserConstraints iso19115:use-constraints
GeographicIdentifierCode iso19115:geographic-identifier-code
GeographicIdentifierAuthorityTitle iso19115:geographic-identifier-authority-title
GeographicIdentifierAuthorityAlternativeTitle iso19115:geographic-identifier-authority-alternative-title
diff --git a/tika-parent/pom.xml b/tika-parent/pom.xml
index 9efe070db6..6d8ad0e9fa 100644
--- a/tika-parent/pom.xml
+++ b/tika-parent/pom.xml
@@ -1724,7 +1724,7 @@
possible'). tika-grpc is excluded: its protobuf generated-sources and compile deps
are not on the aggregate classpath. NOTE: keep this list in sync when modules are
added or removed (see TIKA-4318). -->
- <sourcepath>tika-annotation-processor/src/main/java;tika-app/src/main/java;tika-bundles/tika-bundle-standard/src/main/java;tika-core/src/main/java;tika-detectors/tika-detector-magika/src/main/java;tika-detectors/tika-detector-siegfried/src/main/java;tika-encoding-detectors/tika-encoding-detector-html/src/main/java;tika-encoding-detectors/tika-encoding-detector-icu4j/src/main/java;tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java;tika-encoding-detectors [...]
+ <sourcepath>tika-annotation-processor/src/main/java;tika-app/src/main/java;tika-bundles/tika-bundle-standard/src/main/java;tika-core/src/main/java;tika-detectors/tika-detector-magika/src/main/java;tika-detectors/tika-detector-siegfried/src/main/java;tika-encoding-detectors/tika-encoding-detector-html/src/main/java;tika-encoding-detectors/tika-encoding-detector-icu4j/src/main/java;tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java;tika-encoding-detectors [...]
<subpackages>org.apache.tika</subpackages>
</configuration>
</plugin>
diff --git a/tika-parsers/tika-parsers-extended/tika-parser-scientific-module/src/main/java/org/apache/tika/parser/geoinfo/GeographicInformationParser.java b/tika-parsers/tika-parsers-extended/tika-parser-scientific-module/src/main/java/org/apache/tika/parser/geoinfo/GeographicInformationParser.java
index c006475746..c0c858518f 100644
--- a/tika-parsers/tika-parsers-extended/tika-parser-scientific-module/src/main/java/org/apache/tika/parser/geoinfo/GeographicInformationParser.java
+++ b/tika-parsers/tika-parsers-extended/tika-parser-scientific-module/src/main/java/org/apache/tika/parser/geoinfo/GeographicInformationParser.java
@@ -357,7 +357,7 @@ public class GeographicInformationParser implements Parser {
metadata.add(ISO19115.OTHER_CONSTRAINTS, s.toString());
}
for (Restriction r : c.getUseConstraints()) {
- metadata.add(ISO19115.USER_CONSTRAINTS, r.name());
+ metadata.add(ISO19115.USE_CONSTRAINTS, r.name());
}
}
diff --git a/tika-parsers/tika-parsers-ml/tika-vlm/pom.xml b/tika-parsers/tika-parsers-ml/tika-vlm/pom.xml
index 409e7ebbc0..2824428b1b 100644
--- a/tika-parsers/tika-parsers-ml/tika-vlm/pom.xml
+++ b/tika-parsers/tika-parsers-ml/tika-vlm/pom.xml
@@ -75,6 +75,13 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <!-- vlm-pdf-parsing.json excludes 'pdf-parser', which must resolve (TIKA-3268) -->
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>tika-parser-pdf-module</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
index 6b9d18792c..fb4d30525d 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
@@ -455,9 +455,7 @@ public class RecursiveParserWrapperTest extends TikaTest {
@Test
public void testStreamClosedAfterSpill() throws Exception {
- // When TikaInputStream spills to a temp file (via getPath()/getFile()),
- // the source stream should be closed promptly since all bytes have been
- // consumed and cached - there's no reason to keep it open.
+ // Spill must not close the source early; close() must reach it exactly once.
ParseContext context = new ParseContext();
Metadata metadata = new Metadata();
RecursiveParserWrapper wrapper = new RecursiveParserWrapper(AUTO_DETECT_PARSER, true);
@@ -472,10 +470,11 @@ public class RecursiveParserWrapperTest extends TikaTest {
TikaInputStream tis = TikaInputStream.get(stream);
tis.setCloseShield();
wrapper.parse(tis, handler, metadata, context);
- // Source stream should not be closed after spilling to file
assertEquals(0, stream.counter);
tis.removeCloseShield();
tis.close();
+ // close() must reach the source, spill or no spill
+ assertEquals(1, stream.counter);
}
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
index e1b81fce41..c61f33cc45 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
@@ -253,16 +253,15 @@ public class PDFParser implements Parser, RenderingParser {
}
private PDFParserConfig getConfig(ParseContext parseContext) throws TikaException, IOException {
- // ParseContextConfig.getConfig() handles:
- // 1. Check for PDFParserConfig already in ParseContext (fast path for embedded docs)
- // 2. Check jsonConfigs for "pdf-parser" and deserialize if present
- // 3. Set deserialized config in ParseContext for future lookups
- // 4. Return defaultConfig if no runtime config found
- return ParseContextConfig.getConfig(
+ PDFParserConfig config = ParseContextConfig.getConfig(
parseContext,
"pdf-parser",
PDFParserConfig.class,
defaultConfig);
+ // publish class-keyed for collaborators that read parseContext.get(PDFParserConfig.class),
+ // e.g. PDFBoxRenderer; safe because pdf-parser is this class's sole owner
+ parseContext.set(PDFParserConfig.class, config);
+ return config;
}
private void checkEncryptedPayload(PDDocument pdfDocument,
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/CompressorParser.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/CompressorParser.java
index 004245b4b3..7aa4200ca3 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/CompressorParser.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/CompressorParser.java
@@ -229,6 +229,7 @@ public class CompressorParser implements Parser {
tis.setCloseShield();
CompressorInputStream cis;
+ boolean detected = false;
try {
CompressorParserOptions options =
context.get(CompressorParserOptions.class,
@@ -272,17 +273,18 @@ public class CompressorParser implements Parser {
metadata.set(CONTENT_TYPE, type.toString());
}
}
+ detected = true;
} catch (CompressorException e) {
- tis.removeCloseShield();
if (e.getCause() instanceof MemoryLimitException) {
throw new TikaMemoryLimitException(e.getMessage());
}
throw new TikaException("Unable to uncompress document stream", e);
- } catch (IOException e) {
- //the pack200 workaround (getPath()/Files.newInputStream) can throw IOException;
- //make sure the close shield is removed before propagating
- tis.removeCloseShield();
- throw e;
+ } finally {
+ // covers unchecked escapes too (e.g. InaccessibleObjectException from the
+ // pack200 reflection), which used to leave the shield stuck on
+ if (!detected) {
+ tis.removeCloseShield();
+ }
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-zip-commons/src/main/java/org/apache/tika/detect/zip/DefaultZipContainerDetector.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-zip-commons/src/main/java/org/apache/tika/detect/zip/DefaultZipContainerDetector.java
index 9516f7b8ae..32eaff2fdc 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-zip-commons/src/main/java/org/apache/tika/detect/zip/DefaultZipContainerDetector.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-zip-commons/src/main/java/org/apache/tika/detect/zip/DefaultZipContainerDetector.java
@@ -242,7 +242,7 @@ public class DefaultZipContainerDetector implements Detector {
metadata.set(Zip.DETECTOR_ZIPFILE_OPENED, false);
// If parsing will follow, try salvaging to prepare ZipFile for parser reuse
- if (parseContext.get(ParsingIntent.class) != null) {
+ if (parseContext.get(ParsingIntent.class) == ParsingIntent.WILL_PARSE) {
zip = ZipSalvager.tryToOpenZipFile(tis, metadata);
if (zip != null && LOG.isDebugEnabled()) {
LOG.debug("Successfully salvaged ZIP for parsing");
diff --git a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
index 174d940244..e6b7b9db03 100644
--- a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
+++ b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
@@ -350,9 +350,7 @@ public class TikaAsyncCLI {
if (asyncConfig == null) {
return;
}
- if (asyncConfig.getHandlerType() == BasicContentHandlerFactory.HANDLER_TYPE.TEXT) {
- return;
- }
+ // no TEXT early-return: the default handler is MARKDOWN, not TEXT
ContentHandlerFactory factory = new BasicContentHandlerFactory(asyncConfig.getHandlerType(), -1);
t.getParseContext().set(ContentHandlerFactory.class, factory);
}
diff --git a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
index b90e8af46c..daf4755fc9 100644
--- a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
+++ b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
@@ -101,8 +101,8 @@ public class UnpackExtractor extends ParsingEmbeddedDocumentExtractor {
// Use the delegate parser to parse this entry
try {
- UnpackHandler bytesHandler = context.get(UnpackHandler.class);
tis.setCloseShield();
+ UnpackHandler bytesHandler = context.get(UnpackHandler.class);
if (bytesHandler != null) {
parseWithBytes(tis, handler, metadata, context);
} else {
@@ -139,6 +139,9 @@ public class UnpackExtractor extends ParsingEmbeddedDocumentExtractor {
translated = Files.createTempFile("tika-tmp-", ".bin");
try (OutputStream os = Files.newOutputStream(translated)) {
EMBEDDED_STREAM_TRANSLATOR.translate(tis, metadata, os);
+ } finally {
+ // translate() drains to EOF; without this the parse below sees an empty stream
+ tis.rewind();
}
}
parse(tis, handler, metadata, context);
diff --git a/tika-pipes/tika-pipes-plugins/pom.xml b/tika-pipes/tika-pipes-plugins/pom.xml
index 5f6b34050c..6172faa8d4 100644
--- a/tika-pipes/tika-pipes-plugins/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/pom.xml
@@ -56,10 +56,12 @@
<!-- !!! VERY IMPORTANT -->
<scope>provided</scope>
</dependency>
+ <!-- provided: bundling the plugin/host boundary yields duplicate Class objects -->
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-pipes-api</artifactId>
<version>${project.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
@@ -73,6 +75,45 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.tika</groupId>
+ <artifactId>tika-serialization</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <!-- parent-loaded: on every host via tika-pipes-core; bundling duplicates PipesIteratorBase -->
+ <dependency>
+ <groupId>org.apache.tika</groupId>
+ <artifactId>tika-pipes-iterator-commons</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <!-- logging is host-supplied -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>jcl-over-slf4j</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-core</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j2-impl</artifactId>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-core</artifactId>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/pom.xml
index a2808232c5..70f3fc6b1f 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/pom.xml
@@ -30,8 +30,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -108,9 +106,7 @@
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
- <includeScope>compile</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
+ <includeScope>runtime</includeScope>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/src/main/java/org/apache/tika/pipes/fetcher/atlassianjwt/AtlassianJwtFetcher.java b/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/src/main/java/org/apache/tika/pipes/fetcher/atlassianjwt/AtlassianJwtFetcher.java
index 45cbce9391..bc29143eaa 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/src/main/java/org/apache/tika/pipes/fetcher/atlassianjwt/AtlassianJwtFetcher.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-atlassian-jwt/src/main/java/org/apache/tika/pipes/fetcher/atlassianjwt/AtlassianJwtFetcher.java
@@ -27,6 +27,7 @@ import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
import java.security.NoSuchAlgorithmException;
import java.util.List;
import java.util.Timer;
@@ -237,9 +238,12 @@ public class AtlassianJwtFetcher extends AbstractTikaExtension implements Fetche
private TikaInputStream spool(InputStream content, Metadata metadata) throws IOException {
long start = System.currentTimeMillis();
TemporaryResources tmp = new TemporaryResources();
+ boolean handedOff = false;
+ try {
Path tmpFile = tmp.createTempFile(metadata);
if (config.getMaxSpoolSize() < 0) {
- Files.copy(content, tmpFile);
+ // createTempFile already created the file
+ Files.copy(content, tmpFile, StandardCopyOption.REPLACE_EXISTING);
} else {
try (OutputStream os = Files.newOutputStream(tmpFile)) {
long totalRead = IOUtils.copyLarge(content, os, 0, config.getMaxSpoolSize());
@@ -250,7 +254,15 @@ public class AtlassianJwtFetcher extends AbstractTikaExtension implements Fetche
}
long elapsed = System.currentTimeMillis() - start;
LOG.debug("took {} ms to copy to local tmp file", elapsed);
- return TikaInputStream.get(tmpFile);
+ TikaInputStream tis = TikaInputStream.get(tmpFile, metadata, tmp);
+ handedOff = true;
+ return tis;
+ } finally {
+ // a failed copy must not orphan the temp file
+ if (!handedOff) {
+ tmp.close();
+ }
+ }
}
private void updateMetadata(String url, HttpResponse response, HttpClientContext context,
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-az-blob/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-az-blob/pom.xml
index 72e22d0eeb..a869844670 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-az-blob/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-az-blob/pom.xml
@@ -32,8 +32,6 @@
<properties>
<!-- You want to make sure you don't load competing dependencies in your plugin or your classloading is going
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -43,11 +41,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-serialization</artifactId>
@@ -90,8 +83,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-csv/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-csv/pom.xml
index dc0ae87c11..a01fb2baa5 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-csv/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-csv/pom.xml
@@ -30,16 +30,9 @@
<name>Apache Tika Pipes CSV</name>
<url>https://tika.apache.org/</url>
<properties>
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core,tika-pipes-iterator-commons</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
@@ -70,8 +63,6 @@
<configuration>
<outputDirectory>target/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-es/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-es/pom.xml
index 1170e5e970..58362fd936 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-es/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-es/pom.xml
@@ -35,8 +35,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -81,8 +79,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-file-system/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-file-system/pom.xml
index 5459d99c1e..b14dddb83e 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-file-system/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-file-system/pom.xml
@@ -30,15 +30,8 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-pipes-api</artifactId>
@@ -94,8 +87,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-gcs/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-gcs/pom.xml
index 2d569aa8bb..d17c56e97a 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-gcs/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-gcs/pom.xml
@@ -32,8 +32,6 @@
<properties>
<!-- You want to make sure you don't load competing dependencies in your plugin or your classloading is going
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -43,11 +41,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-serialization</artifactId>
@@ -86,8 +79,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/pom.xml
index 05488ecb39..ac0a5f1517 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/pom.xml
@@ -28,8 +28,6 @@
<artifactId>tika-pipes-google-drive</artifactId>
<name>Apache Tika Pipes Google Drive</name>
<properties>
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
<google-api-services-drive.version>v3-rev20260720-2.0.0</google-api-services-drive.version>
<google-auth-library-oauth2-http.version>1.50.0</google-auth-library-oauth2-http.version>
<google-api-client.version>2.9.0</google-api-client.version>
@@ -154,9 +152,7 @@
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
- <includeScope>compile</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
+ <includeScope>runtime</includeScope>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/src/main/java/org/apache/tika/pipes/fetcher/googledrive/GoogleDriveFetcher.java b/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/src/main/java/org/apache/tika/pipes/fetcher/googledrive/GoogleDriveFetcher.java
index 57b340a32f..7f0e13918e 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/src/main/java/org/apache/tika/pipes/fetcher/googledrive/GoogleDriveFetcher.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-google-drive/src/main/java/org/apache/tika/pipes/fetcher/googledrive/GoogleDriveFetcher.java
@@ -106,7 +106,8 @@ public class GoogleDriveFetcher extends AbstractTikaExtension implements Fetcher
TemporaryResources tmp = new TemporaryResources();
Path tmpPath = tmp.createTempFile(metadata);
FileUtils.copyInputStreamToFile(is, tmpPath.toFile());
- return TikaInputStream.get(tmpPath);
+ // the 3-arg form registers tmp for cleanup
+ return TikaInputStream.get(tmpPath, metadata, tmp);
}
return TikaInputStream.get(is);
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-http/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-http/pom.xml
index 314e2b5567..7e6daf2f33 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-http/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-http/pom.xml
@@ -30,8 +30,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -134,8 +132,6 @@
<configuration>
<outputDirectory>target/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/HttpFetcher.java b/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/HttpFetcher.java
index 3d0c469327..21f43a7048 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/HttpFetcher.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/HttpFetcher.java
@@ -342,6 +342,8 @@ public class HttpFetcher extends AbstractTikaExtension implements Fetcher, Range
private TikaInputStream spool(InputStream content, Metadata metadata) throws IOException {
long start = System.currentTimeMillis();
TemporaryResources tmp = new TemporaryResources();
+ boolean handedOff = false;
+ try {
Path tmpFile = tmp.createTempFile(metadata);
if (httpFetcherConfig.getMaxSpoolSize() < 0) {
Files.copy(content, tmpFile, StandardCopyOption.REPLACE_EXISTING);
@@ -355,7 +357,15 @@ public class HttpFetcher extends AbstractTikaExtension implements Fetcher, Range
}
long elapsed = System.currentTimeMillis() - start;
LOG.debug("took {} ms to copy to local tmp file", elapsed);
- return TikaInputStream.get(tmpFile, metadata, tmp);
+ TikaInputStream tis = TikaInputStream.get(tmpFile, metadata, tmp);
+ handedOff = true;
+ return tis;
+ } finally {
+ // a failed copy must not orphan the temp file
+ if (!handedOff) {
+ tmp.close();
+ }
+ }
}
private void updateMetadata(String url, HttpResponse response, HttpClientContext context, Metadata metadata) {
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/config/HttpFetcherConfig.java b/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/config/HttpFetcherConfig.java
index 27320af480..68efc2f69f 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/config/HttpFetcherConfig.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-http/src/main/java/org/apache/tika/pipes/fetcher/http/config/HttpFetcherConfig.java
@@ -63,7 +63,8 @@ public class HttpFetcherConfig {
private String userAgent;
private String jwtIssuer;
private String jwtSubject;
- private int jwtExpiresInSeconds;
+ // 0 would mint already-expired tokens
+ private int jwtExpiresInSeconds = 3600;
private String jwtSecret;
private String jwtPrivateKeyBase64;
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/pom.xml
index 668c6a86e3..d9a28729f2 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/pom.xml
@@ -32,8 +32,6 @@
<properties>
<!-- You want to make sure you don't load competing dependencies in your plugin or your classloading is going
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -43,11 +41,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-pipes-reporter-commons</artifactId>
@@ -86,8 +79,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/src/main/java/org/apache/tika/pipes/reporter/jdbc/JDBCPipesReporter.java b/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/src/main/java/org/apache/tika/pipes/reporter/jdbc/JDBCPipesReporter.java
index f3239b4461..7fa67f90ac 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/src/main/java/org/apache/tika/pipes/reporter/jdbc/JDBCPipesReporter.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-jdbc/src/main/java/org/apache/tika/pipes/reporter/jdbc/JDBCPipesReporter.java
@@ -95,6 +95,8 @@ public class JDBCPipesReporter extends PipesReporterBase {
}
if (config.reportSql() == null || config.reportSql().isBlank()) {
reportSql = "insert into " + config.tableName() + " (id, status, timestamp) values (?,?,?)";
+ } else {
+ reportSql = config.reportSql();
}
ReportWorker reportWorker = new ReportWorker(config, queue);
reportWorker.init();
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-json/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-json/pom.xml
index 9661e09c7d..4e0eb2f2e4 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-json/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-json/pom.xml
@@ -31,20 +31,15 @@
<url>https://tika.apache.org/</url>
<properties>
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core,tika-pipes-iterator-commons</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
+ <!-- host framework, parent-loaded -->
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-pipes-core</artifactId>
<version>${project.version}</version>
+ <scope>provided</scope>
</dependency>
</dependencies>
<build>
@@ -71,8 +66,6 @@
<configuration>
<outputDirectory>target/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-kafka/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-kafka/pom.xml
index f619bf234e..61d518ab03 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-kafka/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-kafka/pom.xml
@@ -31,8 +31,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -42,11 +40,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
@@ -84,8 +77,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-microsoft-graph/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-microsoft-graph/pom.xml
index 833dc53541..e9fefe7729 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-microsoft-graph/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-microsoft-graph/pom.xml
@@ -35,8 +35,6 @@
<microsoft-kiota.version>1.9.3</microsoft-kiota.version>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencyManagement>
@@ -166,8 +164,6 @@
<configuration>
<outputDirectory>target/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-opensearch/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-opensearch/pom.xml
index 4cca9007da..068f6c1eaa 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-opensearch/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-opensearch/pom.xml
@@ -30,8 +30,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -41,11 +39,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-pipes-reporter-commons</artifactId>
@@ -81,8 +74,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-s3/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-s3/pom.xml
index 4335bce1c1..e469318d24 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-s3/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-s3/pom.xml
@@ -31,8 +31,6 @@
<properties>
<!-- Never include the core artifacts in your plugin lib directory. If you do, it will cause the classloading
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
@@ -42,11 +40,6 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-serialization</artifactId>
@@ -88,8 +81,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-s3/src/main/java/org/apache/tika/pipes/fetcher/s3/S3Fetcher.java b/tika-pipes/tika-pipes-plugins/tika-pipes-s3/src/main/java/org/apache/tika/pipes/fetcher/s3/S3Fetcher.java
index d4a671a927..88748941f6 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-s3/src/main/java/org/apache/tika/pipes/fetcher/s3/S3Fetcher.java
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-s3/src/main/java/org/apache/tika/pipes/fetcher/s3/S3Fetcher.java
@@ -252,6 +252,8 @@ public class S3Fetcher extends AbstractTikaExtension implements Fetcher, RangeFe
tmp = new TemporaryResources();
Path tmpPath = tmp.createTempFile(FilenameUtils.getSuffixFromPath(fetchKey));
Files.copy(s3Object, tmpPath, StandardCopyOption.REPLACE_EXISTING);
+ // Files.copy does not close its input; the success path must
+ s3Object.close();
TikaInputStream tis = TikaInputStream.get(tmpPath, metadata, tmp);
LOGGER.debug("took {} ms to fetch metadata and copy to local tmp file",
System.currentTimeMillis() - start);
diff --git a/tika-pipes/tika-pipes-plugins/tika-pipes-solr/pom.xml b/tika-pipes/tika-pipes-plugins/tika-pipes-solr/pom.xml
index f23a76b45c..421b4cbb25 100644
--- a/tika-pipes/tika-pipes-plugins/tika-pipes-solr/pom.xml
+++ b/tika-pipes/tika-pipes-plugins/tika-pipes-solr/pom.xml
@@ -32,16 +32,9 @@
<properties>
<!-- You want to make sure you don't load competing dependencies in your plugin or your classloading is going
to get messed up when finding your plugins. -->
- <plugin.excluded.artifactIds>tika-core,tika-pipes-api,tika-serialization,tika-plugins-core</plugin.excluded.artifactIds>
- <plugin.excluded.groupIds>org.apache.logging.log4j,org.slf4j</plugin.excluded.groupIds>
</properties>
<dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>tika-pipes-iterator-commons</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>tika-pipes-api</artifactId>
@@ -116,8 +109,6 @@
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
- <excludeArtifactIds>${plugin.excluded.artifactIds}</excludeArtifactIds>
- <excludeGroupIds>${plugin.excluded.groupIds}</excludeGroupIds>
</configuration>
</execution>
</executions>
diff --git a/tika-serialization/src/main/java/org/apache/tika/config/loader/AbstractSpiComponentLoader.java b/tika-serialization/src/main/java/org/apache/tika/config/loader/AbstractSpiComponentLoader.java
index eb0787c711..2338756353 100644
--- a/tika-serialization/src/main/java/org/apache/tika/config/loader/AbstractSpiComponentLoader.java
+++ b/tika-serialization/src/main/java/org/apache/tika/config/loader/AbstractSpiComponentLoader.java
@@ -300,9 +300,14 @@ public abstract class AbstractSpiComponentLoader<T> implements ComponentLoader<T
}
}
+ /**
+ * An unresolvable exclusion stops the world (TIKA-3268): silently ignoring it
+ * would leave a deliberately disabled component enabled.
+ */
@SuppressWarnings("unchecked")
private Set<Class<? extends T>> parseExclusions(JsonNode configNode,
- LoaderContext context) {
+ LoaderContext context)
+ throws TikaConfigException {
Set<Class<? extends T>> exclusions = new HashSet<>();
if (configNode == null || !configNode.isObject()) {
@@ -327,7 +332,12 @@ public abstract class AbstractSpiComponentLoader<T> implements ComponentLoader<T
exclusions.add((Class<? extends T>) clazz);
LOG.debug("Excluding {} from SPI: {}", sectionName, typeName);
} catch (ClassNotFoundException e) {
- LOG.warn("Unknown {} in exclude list: {}", sectionName, typeName);
+ throw new TikaConfigException(
+ "Unknown " + sectionName + " in exclude list: '" + typeName + "'. "
+ + "The exclusion cannot be applied, so the component would stay "
+ + "enabled. In 4.x, components are named by their registered "
+ + "component name (e.g. 'pdf-parser'), not by class name; see the "
+ + "4.x migration guide if this config came from 3.x.", e);
}
}
diff --git a/tika-serialization/src/main/java/org/apache/tika/serialization/ConfigDeserializer.java b/tika-serialization/src/main/java/org/apache/tika/serialization/ConfigDeserializer.java
index 2be514b852..ba6db2a0f4 100644
--- a/tika-serialization/src/main/java/org/apache/tika/serialization/ConfigDeserializer.java
+++ b/tika-serialization/src/main/java/org/apache/tika/serialization/ConfigDeserializer.java
@@ -97,12 +97,11 @@ public class ConfigDeserializer {
// Deserialize and merge with default
T config = JsonMergeUtils.mergeWithDefaults(MAPPER, jsonConfig.json(), configClass, defaultConfig);
- // Cache in resolved configs
+ // Cache per key only. A class-keyed write would leak this config to every
+ // component sharing the class (e.g. the three VLM parsers all bind VLMOCRConfig,
+ // so one provider's base URL and API key would reach the others).
context.setResolvedConfig(configKey, config);
- // Also set in main context so other components can find it via parseContext.get(configClass)
- context.set(configClass, config);
-
return config;
}
diff --git a/tika-serialization/src/test/java/org/apache/tika/config/loader/TikaLoaderTest.java b/tika-serialization/src/test/java/org/apache/tika/config/loader/TikaLoaderTest.java
index 0dc9c36203..d294ac203a 100644
--- a/tika-serialization/src/test/java/org/apache/tika/config/loader/TikaLoaderTest.java
+++ b/tika-serialization/src/test/java/org/apache/tika/config/loader/TikaLoaderTest.java
@@ -19,6 +19,7 @@ package org.apache.tika.config.loader;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.net.URL;
@@ -30,6 +31,7 @@ import org.junit.jupiter.api.Test;
import org.xml.sax.helpers.DefaultHandler;
import org.apache.tika.config.EmbeddedLimits;
+import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
import org.apache.tika.metadata.Metadata;
@@ -252,6 +254,19 @@ public class TikaLoaderTest {
"Should NOT support application/test+minimal");
}
+ /** TIKA-3268: an unresolvable exclusion must fail loudly, not warn. */
+ @Test
+ public void testUnknownExclusionFailsLoudly() throws Exception {
+ URL configUrl =
+ getClass().getResource("/configs/test-default-parser-unknown-exclusion.json");
+ Path configPath = Path.of(configUrl.toURI());
+
+ TikaConfigException e = assertThrows(TikaConfigException.class,
+ () -> TikaLoader.load(configPath).get(Parser.class));
+ assertTrue(e.getMessage().contains("no-such-parser"),
+ "the message must name the unresolvable entry, got: " + e.getMessage());
+ }
+
@Test
public void testDefaultParserWithExclusions() throws Exception {
// Config has "default-parser" with exclude list
diff --git a/tika-serialization/src/test/java/org/apache/tika/serialization/ConfigDeserializerTest.java b/tika-serialization/src/test/java/org/apache/tika/serialization/ConfigDeserializerTest.java
index ead8d15da3..7bc3a426da 100644
--- a/tika-serialization/src/test/java/org/apache/tika/serialization/ConfigDeserializerTest.java
+++ b/tika-serialization/src/test/java/org/apache/tika/serialization/ConfigDeserializerTest.java
@@ -286,4 +286,42 @@ public class ConfigDeserializerTest {
assertTrue(ParseContextConfig.hasConfig(context, "parser-a"));
assertFalse(ParseContextConfig.hasConfig(context, "parser-b"));
}
+
+ /**
+ * Components with distinct config keys may share one config class (the VLM
+ * parsers all bind VLMOCRConfig). Resolving one key must never leak that
+ * config -- base URL, API key -- to another key's component.
+ */
+ @Test
+ public void testSharedConfigClassDoesNotCrossContaminate() throws Exception {
+ ParseContext context = new ParseContext();
+ context.setJsonConfig("provider-a", "{\"name\":\"a\",\"value\":1}");
+ context.setJsonConfig("provider-b", "{\"name\":\"b\",\"value\":2}");
+
+ TestConfig a = ParseContextConfig.getConfig(
+ context, "provider-a", TestConfig.class, new TestConfig());
+ TestConfig b = ParseContextConfig.getConfig(
+ context, "provider-b", TestConfig.class, new TestConfig());
+
+ assertEquals("a", a.getName());
+ assertEquals("b", b.getName());
+ assertEquals(2, b.getValue());
+
+ // and the second lookup of each key returns its own cached instance
+ assertEquals("a", ParseContextConfig.getConfig(
+ context, "provider-a", TestConfig.class, new TestConfig()).getName());
+ }
+
+ /** A programmatic class-keyed override still applies when a key has no JSON config. */
+ @Test
+ public void testProgrammaticClassOverrideStillHonored() throws Exception {
+ ParseContext context = new ParseContext();
+ TestConfig mine = new TestConfig();
+ mine.setName("programmatic");
+ context.set(TestConfig.class, mine);
+
+ TestConfig got = ParseContextConfig.getConfig(
+ context, "no-json-for-this-key", TestConfig.class, new TestConfig());
+ assertEquals("programmatic", got.getName());
+ }
}
diff --git a/tika-serialization/src/test/resources/configs/test-default-parser-unknown-exclusion.json b/tika-serialization/src/test/resources/configs/test-default-parser-unknown-exclusion.json
new file mode 100644
index 0000000000..5b372abcc2
--- /dev/null
+++ b/tika-serialization/src/test/resources/configs/test-default-parser-unknown-exclusion.json
@@ -0,0 +1,9 @@
+{
+ "parsers": [
+ {
+ "default-parser": {
+ "exclude": ["no-such-parser"]
+ }
+ }
+ ]
+}
diff --git a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/TikaResource.java b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/TikaResource.java
index 361304287b..ceb9c72b42 100644
--- a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/TikaResource.java
+++ b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/TikaResource.java
@@ -286,9 +286,7 @@ public class TikaResource {
}
}
- // Create TikaInputStream and spool to temp file immediately.
- // This ensures the data is captured before any other processing
- // and TikaInputStream handles temp file cleanup automatically.
+ // Lazy TikaInputStream: nothing is spooled until a consumer needs a file.
TikaInputStream tis = TikaInputStream.get(fileAtt.getObject(InputStream.class));
boolean handedOff = false;
try {
@@ -442,7 +440,7 @@ public class TikaResource {
/** Parse document and return HTML content. */
@PUT
@Consumes("*/*")
- @Produces("text/html")
+ @Produces("text/html;charset=UTF-8")
@Path("html")
public Response getHtml(final InputStream is, @Context HttpHeaders httpHeaders)
throws IOException {
@@ -452,7 +450,7 @@ public class TikaResource {
/** Parse document and return XML content. */
@PUT
@Consumes("*/*")
- @Produces("text/xml")
+ @Produces("text/xml;charset=UTF-8")
@Path("xml")
public Response getXml(final InputStream is, @Context HttpHeaders httpHeaders)
throws IOException {
@@ -543,7 +541,7 @@ public class TikaResource {
/** Multipart document with optional config; returns HTML. */
@POST
@Consumes("multipart/form-data")
- @Produces("text/html")
+ @Produces("text/html;charset=UTF-8")
@Path("config/html")
public Response postHtml(List<Attachment> attachments, @Context HttpHeaders httpHeaders)
throws IOException, TikaConfigException {
@@ -553,7 +551,7 @@ public class TikaResource {
/** Multipart document with optional config; returns XML. */
@POST
@Consumes("multipart/form-data")
- @Produces("text/xml")
+ @Produces("text/xml;charset=UTF-8")
@Path("config/xml")
public Response postXml(List<Attachment> attachments, @Context HttpHeaders httpHeaders)
throws IOException, TikaConfigException {
diff --git a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/writer/CSVMessageBodyWriter.java b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/writer/CSVMessageBodyWriter.java
index afe4275de8..d99f43d7d1 100644
--- a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/writer/CSVMessageBodyWriter.java
+++ b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/writer/CSVMessageBodyWriter.java
@@ -38,7 +38,7 @@ import org.apache.commons.csv.CSVPrinter;
import org.apache.tika.metadata.Metadata;
@Provider
-@Produces("text/csv")
+@Produces("text/csv;charset=UTF-8")
public class CSVMessageBodyWriter implements MessageBodyWriter<Metadata> {
public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
(tika) 01/02: TIKA-4808: take one, documentation, licenses and other small fixes
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch TIKA-4808-pre-4.0.0-release-fixes
in repository https://gitbox.apache.org/repos/asf/tika.git
commit 7c2ec48c8b77174a2ff21917d0f740136ba255c4
Author: tallison <ta...@apache.org>
AuthorDate: Sun Aug 16 12:57:53 2026 -0400
TIKA-4808: take one, documentation, licenses and other small fixes
---
.github/scripts/check_split_packages.py | 27 +-
CHANGES.txt | 173 ++++-
CONTRIBUTING.md | 4 +-
README.md | 185 +++---
SECURITY.md | 41 +-
.../attachments/advanced/flores-eval-20260320.txt | 10 +-
docs/modules/ROOT/examples/detector-pkcs7.json | 7 +-
docs/modules/ROOT/nav.adoc | 32 +-
.../pages/advanced/charset-detection-design.adoc | 727 +++++++++-----------
.../advanced/charset-detection-eval-20260417.txt | 240 -------
.../charset-detection-eval-v6-no-utf32.txt | 387 -----------
.../ROOT/pages/advanced/charset-detection-eval.txt | 274 --------
.../advanced/charsoup-supported-languages.adoc | 7 +-
.../ROOT/pages/advanced/embedded-documents.adoc | 21 +-
docs/modules/ROOT/pages/advanced/index.adoc | 37 +-
.../integration-testing/run-uat-script.adoc | 31 +-
.../advanced/integration-testing/tika-app.adoc | 26 +-
.../integration-testing/tika-eval-regression.adoc | 117 ++--
.../advanced/integration-testing/tika-server.adoc | 20 +-
.../ROOT/pages/advanced/junk-detection-build.adoc | 555 ++++++++--------
.../ROOT/pages/advanced/junk-detection.adoc | 154 +++--
.../advanced/lang-detection/flores-AUTOMATIC.log | 536 ---------------
.../advanced/lang-detection/flores-SHORT_TEXT.log | 414 ------------
.../advanced/lang-detection/flores-STANDARD.log | 488 --------------
.../advanced/lang-detection/flores200-dev-eval.md | 204 ------
.../lang-detection/language-drop-decisions.md | 175 -----
.../short-text-language-decisions.md | 384 -----------
.../advanced/lang-detection/supported-languages.md | 240 -------
.../pages/advanced/language-detection-build.adoc | 322 ++++-----
.../ROOT/pages/advanced/language-detection.adoc | 266 +++-----
.../ROOT/pages/advanced/local-vlm-server.adoc | 17 +-
docs/modules/ROOT/pages/advanced/robustness.adoc | 136 ++--
.../ROOT/pages/advanced/setting-limits.adoc | 464 +++++--------
docs/modules/ROOT/pages/advanced/spooling.adoc | 61 +-
.../modules/ROOT/pages/advanced/zip-detection.adoc | 20 +-
.../ROOT/pages/configuration/detectors.adoc | 10 +-
.../ROOT/pages/configuration/digesters.adoc | 132 ++--
.../pages/configuration/encoding-detectors.adoc | 178 ++---
docs/modules/ROOT/pages/configuration/index.adoc | 174 +++--
.../ROOT/pages/configuration/metadata-filters.adoc | 254 +++++++
.../configuration/parsers/external-parser.adoc | 38 +-
.../pages/configuration/parsers/pdf-parser.adoc | 12 +-
.../pages/configuration/parsers/tess4j-parser.adoc | 94 +--
.../parsers/tesseract-ocr-parser.adoc | 42 +-
.../pages/configuration/parsers/vlm-parsers.adoc | 64 +-
docs/modules/ROOT/pages/developers/index.adoc | 3 +-
.../ROOT/pages/developers/metadata-keys.adoc | 8 +-
.../ROOT/pages/developers/serialization.adoc | 36 +-
docs/modules/ROOT/pages/faq.adoc | 44 +-
docs/modules/ROOT/pages/index.adoc | 32 +-
.../pages/maintainers/release-guides/docker.adoc | 37 +-
.../pages/maintainers/release-guides/grpc.adoc | 6 +-
.../release-guides/release-artifacts.adoc | 48 +-
.../pages/maintainers/release-guides/tika.adoc | 113 +---
docs/modules/ROOT/pages/maintainers/site.adoc | 50 +-
.../pages/migration-to-4x/chunk-strategies.adoc | 222 +++----
.../pages/migration-to-4x/design-notes-4x.adoc | 61 +-
docs/modules/ROOT/pages/migration-to-4x/index.adoc | 49 +-
.../inference-handler-requirements.adoc | 295 +++------
.../pages/migration-to-4x/metadata-changes-4x.adoc | 133 ++--
.../migration-to-4x/migrating-tika-server-4x.adoc | 170 ++---
.../pages/migration-to-4x/migrating-to-4x.adoc | 120 +++-
.../pages/migration-to-4x/serialization-4x.adoc | 150 ++---
docs/modules/ROOT/pages/pipes/configuration.adoc | 31 +-
docs/modules/ROOT/pages/pipes/cpu-sizing.adoc | 35 +-
docs/modules/ROOT/pages/pipes/emitters.adoc | 6 +-
docs/modules/ROOT/pages/pipes/getting-started.adoc | 27 +-
docs/modules/ROOT/pages/pipes/index.adoc | 27 +-
docs/modules/ROOT/pages/pipes/iterators.adoc | 8 +-
docs/modules/ROOT/pages/pipes/parse-modes.adoc | 4 +
.../ROOT/pages/pipes/plugins/atlassian-jwt.adoc | 18 +-
docs/modules/ROOT/pages/pipes/plugins/azblob.adoc | 25 +-
docs/modules/ROOT/pages/pipes/plugins/csv.adoc | 25 +-
.../ROOT/pages/pipes/plugins/elasticsearch.adoc | 85 ++-
.../ROOT/pages/pipes/plugins/filesystem.adoc | 36 +-
docs/modules/ROOT/pages/pipes/plugins/gcs.adoc | 10 +-
.../ROOT/pages/pipes/plugins/google-drive.adoc | 8 +-
docs/modules/ROOT/pages/pipes/plugins/http.adoc | 27 +-
docs/modules/ROOT/pages/pipes/plugins/index.adoc | 3 +
docs/modules/ROOT/pages/pipes/plugins/jdbc.adoc | 44 +-
docs/modules/ROOT/pages/pipes/plugins/json.adoc | 9 +-
docs/modules/ROOT/pages/pipes/plugins/kafka.adoc | 62 +-
.../ROOT/pages/pipes/plugins/microsoft-graph.adoc | 16 +-
.../ROOT/pages/pipes/plugins/opensearch.adoc | 84 ++-
docs/modules/ROOT/pages/pipes/plugins/s3.adoc | 49 +-
docs/modules/ROOT/pages/pipes/plugins/solr.adoc | 60 +-
.../ROOT/pages/pipes/plugins/writing-a-plugin.adoc | 387 +++++++++++
.../ROOT/pages/pipes/shared-server-mode.adoc | 66 +-
docs/modules/ROOT/pages/pipes/timeouts.adoc | 171 ++---
docs/modules/ROOT/pages/pipes/troubleshooting.adoc | 54 +-
docs/modules/ROOT/pages/pipes/unpack-config.adoc | 101 +--
docs/modules/ROOT/pages/roadmap.adoc | 47 +-
docs/modules/ROOT/pages/security.adoc | 136 ++--
docs/modules/ROOT/pages/using-tika/cli/index.adoc | 433 ++++++------
docs/modules/ROOT/pages/using-tika/docker.adoc | 180 +++++
docs/modules/ROOT/pages/using-tika/grpc/index.adoc | 242 +++----
docs/modules/ROOT/pages/using-tika/index.adoc | 78 +--
.../pages/using-tika/java-api/getting-started.adoc | 98 +--
.../ROOT/pages/using-tika/java-api/index.adoc | 198 +++---
.../ROOT/pages/using-tika/server/index.adoc | 658 ++++++++++---------
docs/modules/ROOT/pages/using-tika/server/tls.adoc | 705 +++++++-------------
llms.txt | 6 +-
pom.xml | 8 +-
.../src/main/appended-resources/META-INF/LICENSE | 68 ++
tika-app/src/main/assembly/assembly.xml | 8 +
.../main/java/org/apache/tika/cli/AsyncHelper.java | 4 +-
.../src/main/java/org/apache/tika/cli/TikaCLI.java | 4 +-
.../test/resources/configs/config-template.json | 1 -
tika-bom/pom.xml | 135 +++-
.../java/org/apache/tika/metadata/Metadata.java | 5 +-
.../main/java/org/apache/tika/metadata/PDF.java | 2 +-
.../main/java/org/apache/tika/metadata/PST.java | 2 +-
.../main/java/org/apache/tika/metadata/XMPPDF.java | 2 +
.../tika/metadata/metadata-migration-3x-4x.json | 9 +-
.../filter/LegacyKeyMigrationFilterTest.java | 37 ++
.../org/apache/tika/pipes/ExternalTestBase.java | 6 +-
.../pipes/filesystem/FileSystemFetcherTest.java | 10 +-
.../tika/pipes/filesystem/HandlerTypeTest.java | 10 +-
.../tika/pipes/ignite/IgniteConfigStoreTest.java | 10 +-
.../src/main/appended-resources/META-INF/LICENSE | 13 +
.../tika-eval-app/src/main/assembly/assembly.xml | 8 +
.../core/tokens/tools/CommonTokenGenerator.java | 4 +-
tika-grpc/src/main/assembly/assembly.xml | 8 +
.../apache/tika/pipes/grpc/TikaGrpcServerImpl.java | 38 +-
tika-grpc/src/main/proto/tika.proto | 2 +-
...PipesBiDirectionalStreamingIntegrationTest.java | 6 +-
.../tika/pipes/grpc/TikaGrpcConcurrencyTest.java | 6 +-
.../apache/tika/pipes/grpc/TikaGrpcServerTest.java | 30 +-
.../pipes/opensearch/tests/OpenSearchTest.java | 6 +-
.../resources/opensearch/plugins-template.json | 6 +-
.../opensearch/tika-config-opensearch.json | 6 +-
.../apache/tika/metadata/metadata-key-fields.json | 5 +-
.../org/apache/tika/metadata/metadata-keys.json | 5 +-
.../org/apache/tika/metadata/migration-overlay.tsv | 13 +
.../detect}/SparseLatinVcardRegressionTest.java | 17 +-
.../org/apache/tika/parser/crypto/Pkcs7Parser.java | 14 +-
.../apache/tika/parser/html/HtmlParserTest.java | 6 -
.../java/org/apache/tika/parser/pdf/PDFParser.java | 1 +
.../org/apache/tika/parser/xmp/XmpExtractor.java | 2 +-
.../configs/config-content-only-default.json | 1 -
.../configs/config-content-only-dynamic.json | 1 -
.../test/resources/configs/config-template.json | 1 -
.../org/apache/tika/client/HttpClientFactory.java | 7 +-
.../tika/pipes/core/PerClientServerManager.java | 10 +-
.../org/apache/tika/pipes/core/PipesConfig.java | 35 +-
.../tika/pipes/core/SharedServerManager.java | 2 +-
.../core/PerClientServerManagerSizingTest.java | 3 +-
.../tika/pipes/core/TikaPipesConfigTest.java | 24 +
.../src/main/assembly/assembly.xml | 8 +
.../apache/tika/pipes/fork/PipesForkParser.java | 33 +-
.../test/resources/configs/tika-config-basic.json | 1 -
.../configs/tika-config-shared-server.json | 3 +-
tika-pipes/tika-pipes-plugins/pom.xml | 18 +
.../tika-pipes-atlassian-jwt/pom.xml | 9 +
.../src/main/appended-resources/META-INF/LICENSE | 59 ++
.../src/main/assembly/assembly.xml | 10 +-
.../atlassianjwt/AtlassianJwtFetcher.java.backup | 351 ----------
.../src/main/resources/plugin.properties | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 55 ++
.../src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika-pipes-csv/src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 29 +
.../tika-pipes-es/src/main/assembly/assembly.xml | 10 +-
.../apache/tika/pipes/emitter/es/ESEmitter.java | 2 +-
.../tika/pipes/emitter/es/HttpClientConfig.java | 14 +-
.../tika/pipes/reporter/es/ESPipesReporter.java | 2 +-
.../src/main/resources/plugin.properties | 2 +-
.../resources/config-examples/es-pipeline.json | 2 +-
.../resources/config-examples/es-reporter.json | 2 +-
.../src/main/assembly/assembly.xml | 10 +-
.../tika/pipes/emitter/fs/FileSystemEmitter.java | 10 +
.../pipes/emitter/fs/FileSystemEmitterConfig.java | 7 +
.../src/main/resources/plugin.properties | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 127 ++++
.../tika-pipes-gcs/src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika-pipes-google-drive/pom.xml | 9 +
.../src/main/appended-resources/META-INF/LICENSE | 119 ++++
.../src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika-pipes-http/src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika/pipes/fetcher/http/HttpFetcherTest.java | 6 +-
.../tika-pipes-jdbc/src/main/assembly/assembly.xml | 10 +-
.../tika/pipes/emitter/jdbc/JDBCEmitterConfig.java | 24 +-
.../src/main/resources/plugin.properties | 2 +-
.../resources/config-examples/jdbc-pipeline.json | 2 +-
.../resources/config-examples/jdbc-reporter.json | 2 +-
.../tika-pipes-json/src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika-pipes-plugins/tika-pipes-kafka/pom.xml | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 33 +
.../src/main/assembly/assembly.xml | 10 +-
.../tika/pipes/emitter/kafka/KafkaEmitter.java | 1 +
.../pipes/emitter/kafka/KafkaEmitterConfig.java | 34 +-
.../src/main/resources/plugin.properties | 2 +-
.../tika-pipes-microsoft-graph/pom.xml | 9 +
.../src/main/appended-resources/META-INF/LICENSE | 136 ++++
.../src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 29 +
.../src/main/assembly/assembly.xml | 10 +-
.../pipes/emitter/opensearch/HttpClientConfig.java | 14 +-
.../emitter/opensearch/OpenSearchEmitter.java | 1 +
.../reporter/opensearch/HttpClientConfig.java | 14 +-
.../opensearch/OpenSearchPipesReporter.java | 1 +
.../opensearch/OpenSearchReporterFactory.java | 2 +-
.../src/main/resources/plugin.properties | 2 +-
.../config-examples/opensearch-pipeline.json | 2 +-
.../config-examples/opensearch-reporter.json | 2 +-
.../src/main/appended-resources/META-INF/LICENSE | 25 +
.../tika-pipes-s3/src/main/assembly/assembly.xml | 10 +-
.../apache/tika/pipes/emitter/s3/S3Emitter.java | 3 -
.../tika/pipes/emitter/s3/S3EmitterConfig.java | 29 +-
.../src/main/resources/plugin.properties | 2 +-
.../resources/config-examples/s3-pipeline.json | 1 -
.../tika-pipes-plugins/tika-pipes-solr/pom.xml | 4 +-
.../src/main/appended-resources/META-INF/LICENSE | 93 +++
.../tika-pipes-solr/src/main/assembly/assembly.xml | 10 +-
.../src/main/resources/plugin.properties | 2 +-
.../config-examples/solr-pipes-iterator.json | 3 +
.../customocr/tika-config-inline.json | 2 +-
.../customocr/tika-config-rendered.json | 6 +-
.../sample-configs/grobid/tika-config.json | 2 +-
.../sample-configs/vision/vlm-claude.json | 14 +-
.../sample-configs/vision/vlm-gemini.json | 12 +-
.../sample-configs/vision/vlm-openai.json | 14 +-
.../tika/server/core/resource/TikaResource.java | 14 +-
.../src/main/appended-resources/META-INF/LICENSE | 729 +++++++++++++++++++++
.../src/main/assembly/assembly.xml | 8 +
232 files changed, 7520 insertions(+), 9030 deletions(-)
diff --git a/.github/scripts/check_split_packages.py b/.github/scripts/check_split_packages.py
index 92e6e1313c..c1c23dba6c 100644
--- a/.github/scripts/check_split_packages.py
+++ b/.github/scripts/check_split_packages.py
@@ -23,22 +23,47 @@ banDuplicateClasses) cannot express.
Scope: main sources only (src/main/java) -- that is what JPMS cares about;
test sources live in the unnamed module and are intentionally excluded.
+Generated code counts too: protoc output ships in the module's jar, so a .proto
+whose `option java_package` names another module's package is a split package
+even though no such directory exists in src/main/java. Those classes land under
+target/, which is pruned here and would need a build to see, so the proto option
+is read straight from src/main/proto instead (TIKA-4808).
+
Usage: python3 .github/scripts/check_split_packages.py [repo_root]
Exit: 0 = no split packages, 1 = split package(s) found.
"""
import os
+import re
import sys
import collections
PRUNE = {"target", ".git", ".local_m2_repo", "node_modules", ".mvn"}
+JAVA_PACKAGE_OPTION = re.compile(
+ r'^\s*option\s+java_package\s*=\s*"([^"]+)"\s*;', re.M)
+
def main() -> int:
root = os.path.abspath(sys.argv[1] if len(sys.argv) > 1 else ".")
pkg_to_modules = collections.defaultdict(set)
- for dirpath, dirnames, _ in os.walk(root):
+ for dirpath, dirnames, files in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in PRUNE]
norm = dirpath.replace(os.sep, "/")
+
+ if norm.endswith("/src/main/proto") or "/src/main/proto/" in norm + "/":
+ marker = os.sep + os.path.join("src", "main", "proto")
+ idx = dirpath.find(marker)
+ if idx > 0:
+ module = os.path.relpath(dirpath[:idx], root).replace(os.sep, "/")
+ for f in files:
+ if not f.endswith(".proto"):
+ continue
+ with open(os.path.join(dirpath, f), encoding="utf-8") as fh:
+ m = JAVA_PACKAGE_OPTION.search(fh.read())
+ if m:
+ pkg_to_modules[m.group(1)].add(module)
+ continue
+
if not norm.endswith("/src/main/java"):
continue
src_root = dirpath
diff --git a/CHANGES.txt b/CHANGES.txt
index d736d877e8..ddf54b492c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -115,14 +115,181 @@ Release 4.0.0 - ???
* tika-server: the tika-server-client module has been removed (TIKA-4809).
+ * Tika 4.x requires Java 17 or later; 3.x built and ran on Java 11. All
+ published artifacts are compiled with --release 17 (TIKA-4685).
+
+ * The core SPI signatures changed. Parser.parse now takes a TikaInputStream
+ instead of an InputStream (there is no InputStream overload),
+ Detector.detect takes (TikaInputStream, Metadata, ParseContext) instead of
+ (InputStream, Metadata), and EmbeddedDocumentExtractor's
+ shouldParseEmbedded/parseEmbedded gained a ParseContext and take a
+ TikaInputStream. Every third-party Parser, Detector or
+ EmbeddedDocumentExtractor implementation must be updated; callers can wrap
+ with TikaInputStream.get(...). The Tika facade (Tika.parse/parseToString)
+ still accepts an InputStream and is unaffected
+ (TIKA-4399, TIKA-4541, TIKA-4569).
+
+ * TikaConfig and the org.apache.tika.config XML-configuration API are
+ removed: TikaConfig, ConfigBase, Field, Param, ParamField,
+ LoadErrorHandler, InitializableProblemHandler, TikaConfigSerializer and
+ TikaTaskTimeout are gone. Use TikaLoader from tika-serialization; see
+ migrating-to-4x.adoc (TIKA-4545, TIKA-4553, TIKA-4565).
+
+ * ForkParser and the entire org.apache.tika.fork package are removed from
+ tika-core. Out-of-process parsing is now provided by PipesForkParser in
+ the new tika-pipes-fork-parser module. tika-app's -f/--fork routes through
+ it, and --fork-timeout is rejected rather than silently ignored
+ (TIKA-4554, TIKA-4571, TIKA-4651).
+
+ * tika-app and tika-server-standard now ship as zip distributions with an
+ adjacent lib/ directory; the published jars are thin launchers and fail
+ with NoClassDefFoundError if run on their own (TIKA-4733).
+
+ * The encoding detectors moved out of parser packages into
+ org.apache.tika.detect.* and into new tika-encoding-detector-* modules:
+ org.apache.tika.parser.txt.{CharsetDetector,CharsetMatch,
+ Icu4jEncodingDetector,UniversalEncodingDetector,BOMDetector,...} are now
+ org.apache.tika.detect.icu4j.*, org.apache.tika.detect.universal.* and
+ org.apache.tika.detect.BOMDetector, and
+ org.apache.tika.parser.html.HtmlEncodingDetector is now
+ org.apache.tika.detect.html.HtmlEncodingDetector.
+ NonDetectingEncodingDetector is removed (TIKA-4685, TIKA-4720).
+
+ * The tika-langdetect-tika module is removed (TikaLanguageDetector,
+ LanguageIdentifier, LanguageProfile, LanguageProfilerBuilder,
+ ProfilingWriter). tika-app, tika-server and tika-eval now bundle the new
+ CharSoup detector (tika-langdetect-charsoup) instead of
+ tika-langdetect-optimaize, so the language reported by default changes
+ (TIKA-4662).
+
+ * SolrJ moves from 8.11.4 to 10.0.0; the Solr fetcher, emitter and pipes
+ iterator no longer support Solr 8 (TIKA-4789).
+
+ * tika-app's batch mode is gone. The -bc/batch directory-to-directory command
+ line (backed by the removed tika-batch module) has no successor flag; use
+ -a/--async, which runs the same work through tika-pipes (TIKA-4340).
+
+ * Metadata's serialVersionUID changed. A Metadata instance serialized by 3.x
+ now fails deserialization with InvalidClassException instead of silently
+ producing an object with a null write limiter that throws on first write
+ (TIKA-4816).
+
+ * PDF: extractIncrementalUpdateInfo now defaults to true (was false), so
+ every PDF parse emits pdf:incremental-update-count and related keys
+ without configuration. parseIncrementalUpdates remains false
+ (TIKA-4354, TIKA-4358).
+
+ * The DOM-based OOXML extractors are removed (XWPFWordExtractorDecorator,
+ XSLFPowerPointExtractorDecorator, POIXMLTextExtractorDecorator,
+ XPSTextExtractor) and with them the OfficeParserConfig keys
+ useSAXDocxExtractor and useSAXPptxExtractor. The SAX extractors are the
+ only implementation (TIKA-4692, TIKA-4708).
+
+ * The MuPDF renderer is removed (org.apache.tika.renderer.pdf.mutool); PDF
+ page rendering for OCR now uses the PDFBox renderer or the new
+ PopplerRenderer (TIKA-4664).
+
+ * Parsers and detectors no longer expose bean setters/getters for their
+ settings. Configuration moves to per-component *Config objects supplied
+ through the ParseContext (e.g. GeoParserConfig, DWGParserConfig,
+ AmazonTranscribeConfig, MagikaDetector/SiegfriedDetector configs)
+ (TIKA-4758).
+
+ * TikaInputStream no longer buffers by default. A stream is consumed in
+ passthrough mode unless enableRewind() is called at position 0;
+ rewind()/getFile()/getPath() after reading without enableRewind() throw
+ instead of silently spooling. Digesters call enableRewind() themselves
+ (TIKA-4618, TIKA-4623).
+
+ * tika-eval-app's command line changed: the FileProfile sub-command is
+ removed, the -bc batch-config option is gone, and extract directories are
+ now named with -e/--extracts (Profile) and -a/--extractsA + -b/--extractsB
+ (Compare); -i/--inputDir, -d/--db, -c/--config, -n/--numWorkers and
+ -m/--maxExtractLength replace the 3.x spellings
+ (TIKA-4342, TIKA-4450, TIKA-4452, TIKA-4507).
+
+ * Audio cover art is now extracted as embedded documents from MP3 (ID3v2
+ APIC/PIC), MP4 (covr), Vorbis and FLAC. Embedded-document counts and
+ /rmeta list lengths for audio files change (TIKA-4801).
+
+ * Additional removals with no direct replacement: the tika-server-eval
+ module (TIKA-4555); SentimentAnalysisParser (TIKA-4574); the
+ tika-age-recogniser module / AgeRecogniser (TIKA-4343);
+ ObjectRecognitionParser and the Tensorflow recognisers/captioners;
+ PooledTimeSeriesParser; org.apache.tika.parser.pdf.AccessChecker
+ (replaced by PDFParserConfig.AccessCheckMode);
+ org.apache.tika.utils.{RereadableInputStream,AnnotationUtils};
+ org.apache.tika.io.{IOUtils,InputStreamFactory};
+ org.apache.tika.sax.DIFContentHandler;
+ org.apache.tika.parser.{AutoDetectParserFactory,ParserFactory};
+ org.apache.tika.parser.internal.Activator; and the jempbox-based
+ JempboxExtractor / XMPMetadataExtractor / pdf.xmpschemas.* classes
+ superseded by the unified XMP extractor (TIKA-4775).
+
NEW FEATURES
- * Content-based detection of ASN.1/DER crypto containers at parse time. An
- opt-in Pkcs7Detector surfaces the subtype at detect() time,
- but must be enabled via configuration (TIKA-1997).
+ * Content-based detection of ASN.1/DER crypto containers. Magic for the
+ PKCS#7/CMS arc ships enabled by default: application/pkcs7-mime gained
+ magic (3.x had globs only), application/pkcs7-signature's magic was
+ broadened across the DER length forms, and application/timestamped-data,
+ application/x-pkcs12, application/x-pkcs7-certificates and
+ application/x-pkcs7-certreqresp gained magic. Pkcs7Parser further refines
+ the smime-type on the output Content-Type. Files that detected as
+ application/octet-stream in 3.x may now detect as a crypto type
+ (TIKA-1997, TIKA-2856).
+
+ * tika-pipes gains three parse modes: NO_PARSE (detect only, no parse),
+ CONTENT_ONLY (emitters write raw content, no metadata envelope) and
+ UNPACK (write embedded bytes out). All three are wired through tika-app
+ and tika-server (TIKA-4631, TIKA-4637, TIKA-4656).
+
+ * New tika-pipes plugins: an Elasticsearch emitter (TIKA-4672), an
+ Atlassian JWT fetcher (TIKA-4604), Google Drive, Microsoft Graph, Azure
+ Blob, JSON and HTTP plugins, and an Apache Ignite ConfigStore for
+ runtime fetcher/emitter configuration (TIKA-4583, TIKA-4587, TIKA-4598).
+
+ * New inference and OCR modules: tika-inference and tika-vlm add
+ vision-language-model parsers (Claude, Gemini, OpenAI) that emit
+ vlm:prompt-tokens / vlm:completion-tokens, and
+ tika-parser-tess4j-module adds in-process Tesseract OCR
+ (TIKA-4665, TIKA-4666, TIKA-4667, TIKA-4690).
+
+ * A Markdown parser with structured, lossless XHTML output, complementing
+ the Markdown content handler (TIKA-4770).
+
+ * New detection: Android binary XML (application/vnd.android.axml,
+ TIKA-4747) and Frictionless Data packages (TIKA-4643); improved mp3/aac
+ (TIKA-4612) and grib (TIKA-4655) detection.
OTHER CHANGES
+ * Dependency upgrades since 4.0.0-beta-1, including Jetty 12.1.12, CXF
+ 4.2.3 and SolrJ 10.0.0 (TIKA-4327).
+
+ * The charset, junk-text and language detection stack was rewritten:
+ language-aware charset detection, a universal junk detector, wider
+ Unicode handling and the new CharSoup language detector
+ (TIKA-4662, TIKA-4671, TIKA-4675, TIKA-4691, TIKA-4719, TIKA-4810).
+
+ * New audio/video metadata: audio:bitrate, audio:is-variable-bitrate,
+ audio:has-drm, audio:channels, video:frame-rate, video:bitrate and MP4
+ sample size; ID3 TCOP and Vorbis COPYRIGHT map to xmpDM:copyright, EXIF
+ GPS altitude maps to geo:alt, and the presentation start of delayed
+ QuickTime timed-metadata tracks is exposed (TIKA-4777, TIKA-4779,
+ TIKA-4780, TIKA-4781, TIKA-4800, TIKA-4802).
+
+ * Parsing and robustness fixes across formats: CHM (TIKA-4783), ID3 UTF-16
+ (TIKA-4784), MPEG2/2.5 Layer III frame sizing (TIKA-4791), .doc empty
+ comments (TIKA-4718), OOXML hyphenation and field-code hyperlinks
+ (TIKA-4646, TIKA-4683), RTF attachments in HTML decapsulation
+ (TIKA-4710), image extraction (TIKA-4736), embedded-file extension
+ calculation (TIKA-4808), and general media-file robustness (TIKA-4812).
+ MAPI properties no longer overwrite better-fitting Dublin Core terms
+ (TIKA-4806). Embedded-file naming was streamlined (TIKA-4689).
+
+ * tika-eval-core is no longer published as a fat jar (TIKA-4414) and
+ tika-grpc no longer shades gRPC (TIKA-4709).
+
* PipesClient/PipesServer IPC now enforces a configurable payload limit
(pipes.maxIpcPayloadBytes, default 100 MB) in both directions. Results
that exceed the limit return PAYLOAD_LIMIT_EXCEEDED instead of causing
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d9ee3ff6f1..91fdbc42e5 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -19,9 +19,7 @@
# Contributing to Apache Tika
-Thank you for your interest in contributing to Apache Tika!
-
-For comprehensive contribution guidelines, please see: **https://tika.apache.org/contribute.html**
+Full guidelines: <https://tika.apache.org/contribute.html>
## Quick Start
diff --git a/README.md b/README.md
index e9cdf4ba47..f8f27b7a5c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,4 @@
-Welcome to Apache Tika <https://tika.apache.org/>
-=================================================
+# Welcome to Apache Tika <https://tika.apache.org/>
[](http://www.apache.org/licenses/LICENSE-2.0)
[](https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/)
@@ -12,8 +11,7 @@ Tika is a project of the [Apache Software Foundation](https://www.apache.org).
Apache Tika, Tika, Apache, the Apache feather logo, and the Apache Tika project logo are trademarks of The Apache Software Foundation.
-Quick Start
-===========
+## Quick Start
**Parse a file in Java:**
@@ -25,10 +23,13 @@ String text = tika.parseToString(new File("document.pdf"));
System.out.println(text);
```
-**From the command line:**
+**From the command line** — unzip `tika-app-<version>.zip` into a directory (the zip
+has no top-level directory of its own) and run from inside it. The jar is a thin
+launcher that loads the parsers from the adjacent `lib/`; on its own it fails with
+`NoClassDefFoundError`.
```bash
-java -jar tika-app-*.jar --text document.pdf
+java -jar tika-app-<version>.jar --text document.pdf
```
**Maven dependency:**
@@ -42,77 +43,79 @@ java -jar tika-app-*.jar --text document.pdf
</dependency>
```
-Getting Started
-===============
-Pre-built binaries of Apache Tika standalone applications are available
-from https://tika.apache.org/download.html . Pre-built binaries of all the
-Tika jars can be fetched from Maven Central or your favourite Maven mirror.
+## Getting Started
-**Tika 2.X and support for Java 8 reached End of Life (EOL) in April, 2025.
-See [Tika Roadmap 2.x, 3.x and beyond](https://cwiki.apache.org/confluence/display/TIKA/Tika+Roadmap+--+2.x%2C+3.x+and+Beyond).**
+Pre-built binaries of the Apache Tika standalone applications are available from
+<https://tika.apache.org/download.html>. Pre-built binaries of all the Tika jars can be
+fetched from Maven Central or your favourite Maven mirror.
+
+Tika 2.x and support for Java 8 reached End of Life (EOL) in April 2025. See the
+[Tika Roadmap](https://cwiki.apache.org/confluence/display/TIKA/Tika+Roadmap+--+2.x%2C+3.x+and+Beyond)
+for the support schedule of each line.
+
+## Building from Source
Tika is based on **Java 17** and uses the [Maven 3](https://maven.apache.org) build system.
-**N.B.** [Docker](https://www.docker.com/products/personal) is used for tests in tika-integration-tests. If Docker is not installed, those tests are skipped.
+The Maven wrapper (`mvnw`) is included in the repository and downloads the correct Maven
+version if needed; on Windows, use `mvnw.cmd` instead.
-To build Tika from source, use the following command in the main directory:
+**N.B.** [Docker](https://www.docker.com/products/personal) is used for tests in
+tika-integration-tests. If Docker is not installed, those tests are skipped.
- ./mvnw clean install
+Build everything from the main directory:
-The Maven wrapper (`mvnw`) is included in the repository and will automatically download
-the correct Maven version if needed. On Windows, use `mvnw.cmd` instead.
+ ./mvnw clean install
-The build consists of a number of components, including a standalone runnable jar that you can use to try out Tika features. You can run it like this:
+That produces a runnable `tika-app` you can use to try out Tika features:
java -jar tika-app/target/tika-app-*.jar --help
-
-To build a specific project (for example, tika-server-standard):
+To build a single project and its dependencies (for example, tika-server-standard):
./mvnw clean install -am -pl :tika-server-standard
-If the ossindex-maven-plugin is causing the build to fail because a dependency
-has now been discovered to have a vulnerability:
+If the ossindex-maven-plugin fails the build because a dependency has since been
+discovered to have a vulnerability:
./mvnw clean install -Dossindex.skip
+### Faster Builds
-Faster Builds
-=============
+* **Fast profile** — `-Pfast` skips tests, checkstyle, and spotless.
+* **Parallel builds** — `-T1C` builds with one thread per CPU core.
+* **[Maven Daemon](https://github.com/apache/maven-mvnd) (`mvnd`)** — keeps a warm JVM
+ running for 2-3x faster rebuilds and is otherwise a drop-in for `mvn`. On macOS:
+ `brew install mvndaemon/tap/mvnd`.
-**Fast profile** - Use `-Pfast` to skip tests, checkstyle, and spotless:
+Combine them for maximum speed during development:
- ./mvnw clean install -Pfast
-
-**Parallel builds** - Add `-T1C` to build with 1 thread per CPU core:
+ mvnd clean install -Pfast -T1C
- ./mvnw clean install -Pfast -T1C
+### Building a Specific Tag
-**Maven Daemon (mvnd)** - Keeps a warm JVM running for 2-3x faster rebuilds:
+To build, say, the 3.0.1 tag:
```bash
-# Install: https://github.com/apache/maven-mvnd
-# macOS: brew install mvndaemon/tap/mvnd
-
-# Use exactly like mvn
-mvnd clean install -Pfast
-mvnd test -pl :tika-core
+git clone https://github.com/apache/tika.git
+cd tika
+git checkout 3.0.1
+./mvnw clean install
```
-**Combine both** for maximum speed during development:
-
- mvnd clean install -Pfast -T1C
+If a new vulnerability has been discovered between the date of the tag and the date you
+are building it, add `-Dossindex.skip`.
+If a local test does not work in your environment, please notify the project at
+dev@tika.apache.org. As an immediate workaround, you can turn off individual tests:
-Reproducible Builds
-===================
+ ./mvnw clean install -Dtest=\!UnpackerResourceTest#testPDFImages
-Apache Tika supports [reproducible builds](https://reproducible-builds.org/). This means
-that building the same source code with the same JDK version should produce
-byte-for-byte identical artifacts, regardless of the build machine or time.
+### Reproducible Builds
-Key configuration:
-- `project.build.outputTimestamp` is set in `tika-parent/pom.xml`
-- All Maven plugins are configured to produce deterministic output
+Apache Tika supports [reproducible builds](https://reproducible-builds.org/): building the
+same source code with the same JDK version produces byte-for-byte identical artifacts,
+regardless of the build machine or time. `project.build.outputTimestamp` is set in
+`tika-parent/pom.xml`, and all Maven plugins are configured to produce deterministic output.
To verify the build plan supports reproducibility:
@@ -125,13 +128,11 @@ To verify two builds produce identical artifacts:
./mvnw clean install -DskipTests
diff -r tika-build-1 ~/.m2/repository/org/apache/tika
+## Maven Dependencies
-Maven Dependencies
-==================
-
-Apache Tika provides *Bill of Material* (BOM) artifact to align Tika module versions and simplify version management.
-To avoid convergence errors in your own project, import this
-bom or Tika's parent pom.xml in your dependency management section.
+Apache Tika provides a *Bill of Materials* (BOM) artifact that aligns Tika module versions.
+Import it (or Tika's parent pom.xml) in your dependency management section to avoid
+convergence errors in your own project.
If you use Apache Maven:
@@ -153,8 +154,8 @@ If you use Apache Maven:
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-parsers-standard-package</artifactId>
- <type>pom</type>
<!-- version not required since BOM included -->
+ <type>pom</type>
</dependency>
</dependencies>
</project>
@@ -171,44 +172,37 @@ dependencies {
}
```
-Migrating to 4.x
-================
-TBD
+## Migrating to 4.x
+
+Upgrading from 3.x requires code and configuration changes: Java 17, `TikaInputStream` in
+the `Parser`/`Detector` SPI, JSON configuration instead of `tika-config.xml`, namespaced
+metadata keys, and Markdown as the default output format. Start with
+[Migrating to Tika 4.x](docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc);
+tika-server users should also read
+[Migrating Tika Server to 4.x](docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc).
-Contributing
-============
-See [CONTRIBUTING.md](CONTRIBUTING.md) and https://tika.apache.org/contribute.html
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) and <https://tika.apache.org/contribute.html>.
[](https://github.com/apache/tika/graphs/contributors)
-Building from a Specific Tag
-============================
-Let's assume that you want to build the 3.0.1 tag:
-```
-0. Download and install hub.github.com
-1. git clone https://github.com/apache/tika.git
-2. cd tika
-3. git checkout 3.0.1
-4. ./mvnw clean install
-```
+## Mailing Lists
+
+* user@tika.apache.org - About using Tika
+* dev@tika.apache.org - About developing Tika
-If a new vulnerability has been discovered between the date of the
-tag and the date you are building the tag, you may need to build with:
+Subscribe by sending a message to `{list}-subscribe@tika.apache.org`.
-```
-4. ./mvnw clean install -Dossindex.skip
-```
+## Issue Tracker
-If a local test is not working in your environment, please notify
- the project at dev@tika.apache.org. As an immediate workaround,
- you can turn off individual tests with e.g.:
+<https://issues.apache.org/jira/browse/TIKA>
-```
-4. ./mvnw clean install -Dossindex.skip -Dtest=\!UnpackerResourceTest#testPDFImages
-```
+## Security
-License (see also LICENSE.txt)
-==============================
+See [SECURITY.md](SECURITY.md) and <https://tika.apache.org/security.html>.
+
+## License (see also LICENSE.txt)
Collective work: Copyright 2011 The Apache Software Foundation.
@@ -220,8 +214,7 @@ Unless required by applicable law or agreed to in writing, software distributed
Apache Tika includes a number of subcomponents with separate copyright notices and license terms. Your use of these subcomponents is subject to the terms and conditions of the licenses listed in the LICENSE.txt file.
-Export Control
-==============
+## Export Control
This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <http://www.wassenaar.org/> for more information.
@@ -229,22 +222,4 @@ The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS
The following provides more details on the included cryptographic software:
-Apache Tika uses the Bouncy Castle generic encryption libraries for extracting text content and metadata from encrypted PDF files. See <http://www.bouncycastle.org/> for more details on Bouncy Castle.
-
-Mailing Lists
-=============
-
-* user@tika.apache.org - About using Tika
-* dev@tika.apache.org - About developing Tika
-
-Subscribe by sending a message to `{list}-subscribe@tika.apache.org`.
-
-Issue Tracker
-=============
-
-https://issues.apache.org/jira/browse/TIKA
-
-Security
-========
-
-See [SECURITY.md](SECURITY.md) and https://tika.apache.org/security.html
+Apache Tika uses the Bouncy Castle generic encryption libraries for extracting text content and metadata from encrypted PDF files. See <http://www.bouncycastle.org/> for more details on Bouncy Castle.
diff --git a/SECURITY.md b/SECURITY.md
index fd3874edb7..4b9dd6970e 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -19,44 +19,35 @@
# Security Policy
-For known security vulnerabilities, see: **https://tika.apache.org/security.html**
-
## Security Model
-Before reporting, please review Tika's security model to understand what is and isn't considered a vulnerability:
-- [Apache Tika Security Model](https://tika.apache.org/security-model.html)
+Before reporting, review Tika's
+[security model](https://tika.apache.org/security-model.html) to understand what is and
+isn't considered a vulnerability.
## Reporting a Vulnerability
-The Apache Tika project takes security seriously. We appreciate your efforts to responsibly disclose your findings.
-
-**Please do NOT report security vulnerabilities through public GitHub or JIRA issues.**
-
-Instead, please report security vulnerabilities privately to the Apache Tika team and to the Apache Security Team:
+**Do NOT report security vulnerabilities through public GitHub or JIRA issues.**
-- **Email**: [security@apache.org](mailto:security@apache.org)
-- **More information**: [Apache Security Team](https://www.apache.org/security/)
+Report them privately to the [Apache Security Team](https://www.apache.org/security/) at
+[security@apache.org](mailto:security@apache.org), including:
-Please include:
- Description of the vulnerability
- Steps to reproduce
- Affected versions
-- Any potential mitigations you've identified
-
+- Any potential mitigations you have identified
## Security Advisories
-Known vulnerabilities are published at:
-- [Apache Tika Security](https://tika.apache.org/security.html)
-- [CVE Database](https://cve.mitre.org/)
+Known vulnerabilities are published on the
+[Apache Tika security page](https://tika.apache.org/security.html) and in the
+[CVE database](https://cve.mitre.org/).
## Supported Versions
-We provide security updates for:
-
-| Version | Supported |
-| ------- | ------------------ |
-| 4.x | :white_check_mark: |
-| 3.x | :white_check_mark: |
-| 2.x | :x: (EOL April 2025) |
-| < 2.0 | :x: |
+| Version | Security updates |
+| ------- | ------------------- |
+| 4.x | Yes |
+| 3.x | Yes |
+| 2.x | No (EOL April 2025) |
+| < 2.0 | No |
diff --git a/docs/modules/ROOT/attachments/advanced/flores-eval-20260320.txt b/docs/modules/ROOT/attachments/advanced/flores-eval-20260320.txt
index 12bd912a97..42e015406b 100644
--- a/docs/modules/ROOT/attachments/advanced/flores-eval-20260320.txt
+++ b/docs/modules/ROOT/attachments/advanced/flores-eval-20260320.txt
@@ -19,10 +19,10 @@ limitations under the License.
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.tika:tika-parent:pom:4.0.0-SNAPSHOT
-[WARNING] 'version' contains an expression but should be a constant. @ org.apache.tika:tika-parent:${revision}, /Users/tallison/Intellij/tika-main-chardet/tika-parent/pom.xml, line 35, column 12
+[WARNING] 'version' contains an expression but should be a constant. @ org.apache.tika:tika-parent:${revision}, <tika-checkout>/tika-parent/pom.xml, line 35, column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.tika:tika:pom:4.0.0-SNAPSHOT
-[WARNING] 'version' contains an expression but should be a constant. @ org.apache.tika:tika-parent:${revision}, /Users/tallison/Intellij/tika-main-chardet/tika-parent/pom.xml, line 35, column 12
+[WARNING] 'version' contains an expression but should be a constant. @ org.apache.tika:tika-parent:${revision}, <tika-checkout>/tika-parent/pom.xml, line 35, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
@@ -38,12 +38,12 @@ limitations under the License.
[INFO] --- exec:3.6.3:java (default-cli) @ tika-langdetect-charsoup ---
CharSoup strategy: STANDARD
Evaluation threads: 12
-Loading test data: /Users/tallison/datasets/flores-200/flores200_dev.tsv
+Loading test data: <datasets>/flores-200/flores200_dev.tsv
Flores-200 mode: normalizing xxx_Yyyy → xxx codes
(multi-script variants kept as xxx_Yyyy separate classes)
Test sentences: 203,381
-Loading CharSoup model: /Users/tallison/datasets/wikipedia-model-v14/langdetect-v14.bin
+Loading CharSoup model: <datasets>/wikipedia-model-v14/langdetect-v14.bin
CharSoup model: 204 classes, 32768 buckets, flags=0xE81, ~8.1 MB heap
Evaluation routes through CharSoupLanguageDetector (script gate + confusable group collapse).
Loading OpenNLP detector(s)...
@@ -571,7 +571,7 @@ pol 93.5% (no misses recorded)
nob 93.7% dan→20, nno→20, swe→1
fas 94.3% mzn→1, eng→1
-Report written to: /Users/tallison/datasets/wikipedia-model-v14/flores-v14-eval.log
+Report written to: <datasets>/wikipedia-model-v14/flores-v14-eval.log
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
diff --git a/docs/modules/ROOT/examples/detector-pkcs7.json b/docs/modules/ROOT/examples/detector-pkcs7.json
deleted file mode 100644
index c4d7c5c9b8..0000000000
--- a/docs/modules/ROOT/examples/detector-pkcs7.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
- "detectors": [
- { "pkcs-7-detector": {} },
- { "default-detector": {} }
- ]
-}
diff --git a/docs/modules/ROOT/examples/detector-pkcs7.json b/docs/modules/ROOT/examples/detector-pkcs7.json
new file mode 120000
index 0000000000..c81050be54
--- /dev/null
+++ b/docs/modules/ROOT/examples/detector-pkcs7.json
@@ -0,0 +1 @@
+../../../../tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/detector-pkcs7.json
\ No newline at end of file
diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc
index a61ad78ee0..b1bb44ef77 100644
--- a/docs/modules/ROOT/nav.adoc
+++ b/docs/modules/ROOT/nav.adoc
@@ -16,24 +16,27 @@
* xref:index.adoc[Home]
* xref:using-tika/index.adoc[Using Tika]
** xref:using-tika/java-api/index.adoc[Java API]
+*** xref:using-tika/java-api/getting-started.adoc[Getting Started with the Java API]
+** xref:using-tika/cli/index.adoc[Command Line]
** xref:using-tika/server/index.adoc[Tika Server]
*** xref:using-tika/server/tls.adoc[TLS/SSL Configuration]
-** xref:using-tika/cli/index.adoc[Command Line]
** xref:using-tika/grpc/index.adoc[gRPC]
+** xref:using-tika/docker.adoc[Running Tika in Docker]
* xref:pipes/index.adoc[Pipes]
** xref:pipes/getting-started.adoc[Getting Started]
+** xref:pipes/configuration.adoc[Pipeline Configuration]
** xref:pipes/fetchers.adoc[Fetchers]
** xref:pipes/emitters.adoc[Emitters]
** xref:pipes/iterators.adoc[Iterators]
** xref:pipes/reporters.adoc[Reporters]
-** xref:pipes/configuration.adoc[Pipeline Configuration]
-** xref:pipes/shared-server-mode.adoc[Shared Server Mode]
** xref:pipes/parse-modes.adoc[Parse Modes]
** xref:pipes/unpack-config.adoc[Extracting Embedded Bytes]
** xref:pipes/timeouts.adoc[Timeouts]
-** xref:pipes/cpu-sizing.adoc[Forked-JVM CPU Sizing]
+** xref:pipes/cpu-sizing.adoc[Forked-JVM CPU and Heap Sizing]
+** xref:pipes/shared-server-mode.adoc[Shared Server Mode]
** xref:pipes/troubleshooting.adoc[Troubleshooting]
** xref:pipes/plugins/index.adoc[Plugins]
+*** xref:pipes/plugins/writing-a-plugin.adoc[Writing a Pipes Plugin]
*** xref:pipes/plugins/filesystem.adoc[File System]
*** xref:pipes/plugins/s3.adoc[Amazon S3]
*** xref:pipes/plugins/gcs.adoc[Google Cloud Storage]
@@ -55,6 +58,10 @@
** xref:configuration/parsers/vlm-parsers.adoc[VLM Parsers (Claude, Gemini, OpenAI)]
** xref:configuration/parsers/external-parser.adoc[External Parser (ffmpeg, exiftool, etc.)]
** xref:configuration/parsers/tess4j-parser.adoc[Tess4J OCR (In-Process, advanced)]
+** xref:configuration/metadata-filters.adoc[Metadata Filters]
+** xref:configuration/detectors.adoc[Detectors]
+** xref:configuration/digesters.adoc[Digesters]
+** xref:configuration/encoding-detectors.adoc[Encoding Detectors]
* xref:migration-to-4x/index.adoc[Migration to 4.x]
** xref:migration-to-4x/migrating-to-4x.adoc[Migration Guide]
** xref:migration-to-4x/migrating-tika-server-4x.adoc[Tika Server Migration]
@@ -64,21 +71,28 @@
** xref:migration-to-4x/chunk-strategies.adoc[Chunk Strategies]
** xref:migration-to-4x/inference-handler-requirements.adoc[Inference Handler Requirements]
* xref:advanced/index.adoc[Advanced]
+** xref:advanced/robustness.adoc[Robustness]
+** xref:advanced/setting-limits.adoc[Setting Limits]
+** xref:advanced/spooling.adoc[TikaInputStream and Spooling]
** xref:advanced/language-detection.adoc[Language Detection]
** xref:advanced/language-detection-build.adoc[Building the Language Detector]
+** xref:advanced/charsoup-supported-languages.adoc[CharSoup Supported Languages]
** xref:advanced/junk-detection.adoc[Text Quality Scoring (Junk Detection)]
** xref:advanced/junk-detection-build.adoc[Building the Junk Detector]
-** xref:advanced/robustness.adoc[Robustness]
-** xref:advanced/setting-limits.adoc[Setting Limits]
-** xref:advanced/spooling.adoc[Spooling]
+** xref:advanced/charset-detection-design.adoc[Charset Detection Pipeline]
+** xref:advanced/zip-detection.adoc[ZIP Detection and Salvaging]
** xref:advanced/embedded-documents.adoc[Embedded Document Metadata]
** xref:advanced/local-vlm-server.adoc[Running a Local VLM Server]
-** xref:advanced/integration-testing/run-uat-script.adoc[Tika-Server REST UAT Script]
+** Integration Testing
+*** xref:advanced/integration-testing/tika-app.adoc[Testing with Tika App]
+*** xref:advanced/integration-testing/tika-server.adoc[Testing with Tika Server]
+*** xref:advanced/integration-testing/run-uat-script.adoc[Tika-Server REST UAT Script]
+*** xref:advanced/integration-testing/tika-eval-regression.adoc[Regression Evaluation with tika-eval]
+* xref:security.adoc[Security]
* xref:developers/index.adoc[Developers]
** xref:developers/serialization.adoc[Serialization and Configuration]
** xref:developers/metadata-keys.adoc[Adding a Metadata Key]
* xref:faq.adoc[FAQ]
-* xref:security.adoc[Security]
* xref:roadmap.adoc[Roadmap]
* xref:maintainers/index.adoc[Maintainers]
** xref:maintainers/site.adoc[Publishing the Site]
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
index 815ed375e4..76e74133a6 100644
--- a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
+++ b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
@@ -19,17 +19,16 @@
:toc:
:toc-placement: left
-Tika 4.x charset detection: byte-bigram NB classifier, structural
-detectors for UTF-16/UTF-32, and a script-aware text-quality arbitrator.
-Design: collect candidates from every layer, then arbitrate by decode
-quality — not first-detector-wins.
+Byte-bigram Naive Bayes classification, structural detectors for
+UTF-16/UTF-32/ISO-2022, and a script-aware text-quality arbitrator. The design
+decision that shapes everything else: collect candidates from every layer, then
+arbitrate by decode quality — not first-detector-wins.
== Pipeline overview
-The default `EncodingDetector` chain is assembled via the Java SPI. All
-results are collected into an `EncodingDetectorContext` on the
-`ParseContext`; when a `MetaEncodingDetector` (currently
-`JunkFilterEncodingDetector`) is present,
+The default `EncodingDetector` chain is assembled via the Java SPI. All results
+are collected into an `EncodingDetectorContext` on the `ParseContext`; when a
+`MetaEncodingDetector` (currently `JunkFilterEncodingDetector`) is present,
`CompositeEncodingDetector` switches into collect-all-then-arbitrate mode.
[cols="1,2,1,2", options="header"]
@@ -39,38 +38,35 @@ results are collected into an `EncodingDetectorContext` on the
| 1
| `BOMDetector`
| `tika-core`
-| Reads the first 4 bytes. Returns a DECLARATIVE result when a
- UTF-8 / UTF-16 / UTF-32 BOM is present.
+| First 4 bytes. DECLARATIVE result on a UTF-8 / UTF-16 / UTF-32 BOM.
+ See <<bom-detector>>.
| 2
| `MetadataCharsetDetector`
| `tika-core`
-| Reads `Content-Type` charset parameter and `Content-Encoding` from
- `Metadata`. Applies WHATWG label normalization (ISO-8859-1 and
- US-ASCII → windows-1252). DECLARATIVE.
+| `Content-Type` charset parameter and `Content-Encoding` from `Metadata`,
+ with WHATWG label normalization. DECLARATIVE. See <<metadata-detector>>.
| 3
| `MojibusterEncodingDetector`
| `tika-encoding-detector-mojibuster`
-| Structural UTF-32 and UTF-16 detection, UTF-8 grammar gate, HTML
- attribute-aware stripping, then a 34-class byte-bigram NB
- classifier. STRUCTURAL for structural hits; STATISTICAL for NB
- predictions. See <<nb-pipeline>>.
+| Structural UTF-32/UTF-16/ISO-2022 detection, UTF-8 grammar gate, HTML
+ stripping, then a 34-class byte-bigram NB classifier. STRUCTURAL for
+ structural hits, STATISTICAL for NB predictions. See <<nb-pipeline>>.
| 4
| `HtmlEncodingDetector`
| `tika-encoding-detector-html`
-| Scans HTML `<meta charset>` / `<meta http-equiv=Content-Type>` tags with a
- fast lenient regex matcher. Returns a DECLARATIVE result. Applies a
- curated subset of WHATWG label aliases (see <<html-charset-aliases>>).
- An alternative, spec-strict implementation — `StandardHtmlEncodingDetector`
- — is available opt-in for users who need the full WHATWG prescan algorithm.
+| `<meta charset>` / `<meta http-equiv=Content-Type>` via a fast lenient regex
+ matcher, resolved through the curated alias table in
+ `TikaHtmlCharsetAliases`. DECLARATIVE. `StandardHtmlEncodingDetector` is
+ available opt-in for the full spec-strict WHATWG prescan.
| 5
| `JunkFilterEncodingDetector`
| `tika-ml-junkdetect`
-| `MetaEncodingDetector`. Arbitrates across all base-detector
- candidates using a script-aware text-quality model. See <<junk-filter>>.
+| `MetaEncodingDetector`. Arbitrates across all base-detector candidates using
+ a script-aware text-quality model. See <<junk-filter>>.
|===
=== EncodingResult and ResultType
@@ -79,295 +75,302 @@ Detectors return `List<EncodingResult>`. Each carries:
* `charset` — the detected `java.nio.charset.Charset`.
* `confidence` — 0.0 to 1.0.
-* `label` — the detector's internal label (may be finer than the Java
- charset name, e.g. `IBM424-ltr`, `UTF-16-BE`).
-* `resultType` — one of `DECLARATIVE`, `STRUCTURAL`, `STATISTICAL`.
-
-In the default chain, JunkFilter arbitrates across all tiers by
-decode-quality scoring. Declared charsets aren't automatically
-authoritative, but a declared charset whose decoding is byte-identical
-to at least one other candidate's wins. See
+* `label` — the detector's internal label, which may be finer than the Java
+ charset name (e.g. `IBM424-ltr`, `UTF-16-BE`).
+* `resultType` — `DECLARATIVE`, `STRUCTURAL`, or `STATISTICAL`.
+
+In the default chain JunkFilter arbitrates across all three tiers by decode
+quality. A declared charset is not automatically authoritative, but one whose
+decoding is byte-identical to at least one other candidate's wins. See
<<opting-out-of-arbitration>> for plain first-match-wins.
+[[bom-detector]]
+== BOMDetector
+
+Reads the first 4 bytes:
+
+[cols="1,1", options="header"]
+|===
+| Byte sequence | Encoding
+
+| `EF BB BF` | UTF-8
+| `FF FE 00 00` | UTF-32-LE
+| `00 00 FE FF` | UTF-32-BE
+| `FF FE` | UTF-16-LE
+| `FE FF` | UTF-16-BE
+|===
+
+Returns DECLARATIVE. The HTML detectors deliberately do not handle BOMs, so
+`BOMDetector` is the sole source of BOM evidence; that lets JunkFilter arbitrate
+when a BOM and a `<meta charset>` tag disagree.
+
+[[metadata-detector]]
+== MetadataCharsetDetector
+
+Reads declarative charset hints from the `Metadata` object before any byte
+analysis:
+
+* `Content-Type` charset parameter (e.g. `text/html; charset=windows-1251`)
+* `Content-Encoding` (used by `RFC822Parser` and similar MIME-aware parsers)
+
+`ISO-8859-1` and `US-ASCII` are normalized to `windows-1252`, because browsers
+and the HTML5 spec treat them as aliases for it in practice. Returns
+DECLARATIVE.
+
[[nb-pipeline]]
== MojibusterEncodingDetector
-The pipeline runs layers in order and returns the first confident
-structural answer when one fires; otherwise it falls through to NB.
+Layers run in order; the first confident structural answer short-circuits,
+otherwise the probe falls through to NB.
=== Layer 1 — UTF-32 structural (`WideUnicodeDetector`)
-Every 4-byte group is interpreted as a 32-bit codepoint in both BE and
-LE order and checked for Unicode validity (0x000000 – 0x10FFFF,
-excluding surrogates). Valid codepoints occupy only ~0.004% of the
-32-bit space, so non-UTF-32 bytes almost always produce out-of-range
-values within the first 8 bytes.
+Every 4-byte group is read as a 32-bit codepoint in both BE and LE order and
+checked for Unicode validity (0x000000–0x10FFFF, surrogates excluded). Valid
+codepoints occupy ~0.004% of the 32-bit space, so non-UTF-32 bytes almost always
+produce an out-of-range value within the first 8 bytes.
=== Layer 2 — UTF-16 specialist (`Utf16SpecialistEncodingDetector`)
-A maxent classifier trained on stride-2 column-histogram features.
-The model sees "byte distribution at even positions" vs "byte
-distribution at odd positions" — the structural property that
-distinguishes UTF-16 from legacy byte encodings.
-
-An external column-asymmetry evidence gate blocks the specialist on
-probes that look unambiguously single-stream (e.g. legacy CJK). When
-the specialist has a confident LE-vs-BE winner, the pipeline
-short-circuits.
+A maxent classifier over stride-2 column-histogram features: byte distribution
+at even positions versus odd positions, the structural property that separates
+UTF-16 from legacy byte encodings. An external column-asymmetry evidence gate
+blocks the specialist on probes that look unambiguously single-stream (e.g.
+legacy CJK). A confident LE-vs-BE winner short-circuits the pipeline.
-Byte bigrams (stride-1) cannot reliably discriminate UTF-16 of CJK
-content from legacy CJK encodings, because CJK characters encode to
-byte pairs that alias common ASCII byte patterns (e.g. U+6572 is
-`72 65` in UTF-16-LE — same bytes as ASCII "re"). That's why UTF-16
-lives outside the main NB classifier.
+UTF-16 lives outside the main NB classifier because stride-1 byte bigrams cannot
+discriminate UTF-16 CJK content from legacy CJK encodings: CJK characters encode
+to byte pairs that alias common ASCII patterns (U+6572 is `72 65` in UTF-16-LE —
+the same bytes as ASCII "re").
=== Layer 3 — UTF-8 grammar gate (`StructuralEncodingRules.checkUtf8`)
-Walks the probe looking for valid UTF-8 lead + continuation byte
-sequences. Three outcomes:
-
-* `LIKELY_UTF8` — grammar is valid and at least one complete multi-byte
- sequence is present. Pipeline emits UTF-8 as a STRUCTURAL candidate
- alongside NB's output (safety net for short probes where NB could
- be fooled by a single coincidental bigram).
-* `NOT_UTF8` — grammar is violated. UTF-8 is filtered out of NB's
- output as a post-pass disqualifier.
-* `AMBIGUOUS` — no complete multi-byte sequence (pure ASCII, or only
- a truncated lead at probe-end). No emission.
-
-=== ISO-2022-JP/KR/CN structural detection (pure-ASCII branch)
-
-ISO-2022 encodings are 7-bit and escape-based (`ESC $ B`, `ESC $ ) C`, …),
-so they carry no high bytes and are invisible to the byte-bigram
-classifier; without a structural check a real ISO-2022-JP page would fall
-through to the windows-1252 default and decode to gibberish. On a
-pure-ASCII probe — the only place ISO-2022 can occur — the pipeline scans
-for the ISO-2022 designation escape and, if found, *verifies* by decoding:
-the result must contain real CJK at a near-zero replacement rate. The
-verify rejects a stray `ESC $` in ordinary ASCII (which yields no CJK).
-On success an ISO-2022-JP/KR/CN STRUCTURAL candidate is emitted. High-byte
-binary that happens to contain an escape sequence never reaches this
-check — it fails the pure-ASCII gate and takes the normal NB path, so it
-cannot trigger a false ISO-2022 detection.
+Walks the probe looking for valid UTF-8 lead + continuation sequences:
+
+* `LIKELY_UTF8` — grammar valid and at least one complete multi-byte sequence.
+ UTF-8 is emitted as a STRUCTURAL candidate alongside NB's output, a safety net
+ for short probes where NB could be fooled by one coincidental bigram.
+* `NOT_UTF8` — grammar violated. UTF-8 is filtered out of NB's output.
+* `AMBIGUOUS` — no complete multi-byte sequence (pure ASCII, or a truncated lead
+ at probe end). No emission.
+
+=== ISO-2022-JP/KR/CN structural detection
+
+ISO-2022 encodings are 7-bit and escape-based (`ESC $ B`, `ESC $ ) C`, …), so
+they carry no high bytes and are invisible to the byte-bigram classifier;
+without a structural check a real ISO-2022-JP page would fall through to the
+windows-1252 default and decode to gibberish.
+
+On a pure-ASCII probe — the only place ISO-2022 can occur — the pipeline scans
+for the designation escape and then *verifies* by decoding: the result must
+contain real CJK at a near-zero replacement rate. That rejects a stray `ESC $`
+in ordinary ASCII. High-byte binary containing an escape sequence fails the
+pure-ASCII gate and takes the normal NB path, so it cannot trigger a false
+ISO-2022 detection.
=== Layer 4 — HTML stripping (content-type aware)
-When the probe looks like HTML/XML (explicit content-type or unknown),
-`HtmlByteStripper` performs a byte-level state-machine strip to leave
-just content bytes for NB feature extraction. Optimizations:
+When the probe looks like HTML/XML (explicit content type, or unknown),
+`HtmlByteStripper` performs a byte-level state-machine strip so NB sees content
+bytes rather than markup:
-* **Script / style / comment bodies** are dropped entirely — they
- contribute no natural-language signal.
+* **Script / style / comment bodies** are dropped entirely — no natural-language
+ signal.
* **Text-bearing attribute values** (`alt`, `title`, `placeholder`,
- `aria-label`, `summary`, `label`) are preserved, separated by
- spaces. These attributes carry user-visible text in the document's
- charset and provide meaningful signal on pages whose body is sparse
- (Arabic forums with navigation in `title=...`, CJK sites with
- screen-reader `aria-label=...`, etc.).
-* **Tag-count backoff** — if stripping produces zero well-formed tags,
- the original bytes are used (the probe wasn't really HTML; common
- for plain text with stray `<` bytes or EBCDIC-encoded content).
+ `aria-label`, `summary`, `label`) are kept, space-separated. They carry
+ user-visible text in the document's charset and are often the only signal on
+ pages with a sparse body — Arabic forums with navigation in `title=...`, CJK
+ sites with screen-reader `aria-label=...`.
+* **Tag-count backoff** — if stripping yields zero well-formed tags the original
+ bytes are used. The probe was not really HTML; common for plain text with
+ stray `<` bytes, or EBCDIC content.
=== Layer 5 — Naive Bayes byte-bigram classifier
-34 classes: CJK multibyte (Big5-HKSCS, EUC-JP, GB18030, Shift_JIS,
-x-EUC-TW, x-windows-949), EBCDIC family (IBM420/424-ltr/rtl, IBM500,
-IBM1047), DOS OEM (IBM850/852/855/866), Cyrillic (KOI8-R, KOI8-U),
-Windows single-byte (1250-1258, 874), ISO-8859-2/3/16, Mac (x-MacRoman,
-x-mac-cyrillic), and UTF-8.
+34 classes: CJK multibyte (Big5-HKSCS, EUC-JP, GB18030, Shift_JIS, x-EUC-TW,
+x-windows-949), EBCDIC (IBM420/424-ltr/rtl, IBM500, IBM1047), DOS OEM
+(IBM850/852/855/866), Cyrillic (KOI8-R, KOI8-U), Windows single-byte (1250–1258,
+874), ISO-8859-2/3/16, Mac (x-MacRoman, x-mac-cyrillic), and UTF-8.
-Features are **stride-1 byte bigrams** — for probe bytes `b[0..N]`,
-feature value for bigram `b[i] << 8 | b[i+1]` counts how many times
-it occurs. No hashing: each bigram is directly an index into a
-65 536-entry table.
+Features are **stride-1 byte bigrams**: the value for bigram
+`b[i] << 8 | b[i+1]` is its occurrence count. No hashing — each bigram indexes
+directly into a 65 536-entry table.
==== Per-class vocabulary
-Each class has its own vocabulary sized to the top bigrams covering
-99.9% of its training marginal distribution. Latin/EBCDIC classes
-need ~4-9 K bigrams; CJK classes ~15-37 K. Uniform top-K would
-either waste slots on SBCS classes (noise) or starve CJK classes.
+Each class gets its own vocabulary, sized to the top bigrams covering 99.9% of
+its training marginal. In the shipped model that is ~3.7–10.5 K bigrams for the
+single-byte and EBCDIC classes and ~13–35 K for UTF-8 and the CJK classes. A
+uniform top-K would either waste slots on SBCS classes or starve CJK classes.
-==== Per-class Laplace smoothing α
+==== Per-class Laplace smoothing
-`α(c) = α_base / V(c)`. Peaky Latin classes get tight smoothing
-(observed frequencies are trustworthy); diffuse CJK classes get
-flatter smoothing (no single bigram is strong evidence).
+`α(c) = α_base / V(c)`. Peaky Latin classes get tight smoothing — their
+observed frequencies are trustworthy; diffuse CJK classes get flatter smoothing,
+because no single bigram is strong evidence there.
==== Shared reference denominator
-The Laplace denominator uses `N_ref = max(total_c)` across classes,
-not per-class totals. Without this, classes with smaller training
-corpora (Mac variants trained on 10M bigrams vs 30M for the main
-classes) end up with milder unseen-log-probabilities, giving them a
-systematic free-lift on sparse-evidence probes. Shared `N_ref`
-eliminates the disparity while preserving relative within-class
-frequencies.
+The Laplace denominator uses `N_ref = max(total_c)` across classes rather than
+per-class totals. Otherwise classes with smaller training corpora (Mac variants
+at 10M bigrams vs 30M for the main classes) get milder unseen log-probabilities
+and a systematic free lift on sparse-evidence probes.
==== IDF reweighting
-A global `float[65_536]` IDF table multiplies each bigram's contribution
-to every class score:
-`idf[b] = log((C + 1) / (df[b] + 1))` where `df[b]` is the number of
-classes in which bigram `b` has non-zero training count. Bigrams
-appearing in every class (common ASCII) contribute near zero;
-class-specific bigrams (CJK signal bytes, EBCDIC-unique patterns)
-carry full weight. Bigrams with IDF quantized to zero skip the
-inner class loop entirely.
+A global `float[65_536]` IDF table multiplies each bigram's contribution to every
+class score: `idf[b] = log((C + 1) / (df[b] + 1))`, where `df[b]` is the number
+of classes in which bigram `b` has a non-zero training count. Bigrams in every
+class (common ASCII) contribute near zero; class-specific bigrams (CJK signal
+bytes, EBCDIC-unique patterns) carry full weight. Bigrams whose IDF quantizes to
+zero skip the inner class loop entirely.
==== Int8 quantization
-Weights and IDF are int8-quantized with per-class scales. Inner loop
-is pure integer multiply-accumulate; a single per-class dequant
-multiplication happens at end-of-probe. Memory: 2 MB on disk (NBB3
-format), ~2 MB in memory. Accuracy: byte-exact-match with float at
-every probe length we've measured.
+Weights and IDF are int8-quantized with per-class scales. The inner loop is pure
+integer multiply-accumulate; one per-class dequant multiplication happens at
+end-of-probe. Accuracy is byte-exact against float at every probe length
+measured.
==== Post-NB fallbacks
-* **Empty / near-empty probes (< 2 bytes)** → windows-1252 @ 0.1
- confidence. WHATWG default; never returns empty result.
-* **Pure ASCII probes** (no bytes ≥ 0x80, no nulls) → ISO-2022 structural
- detection first (see above); otherwise windows-1252. Bigram NB cannot
- discriminate Latin code pages on pure-ASCII content; return the
- HTML5-canonical answer directly.
-* **Latin-sibling → windows-1252 rewrite** — on low-evidence probes
- (< 5 high bytes), if the top NB candidate is a non-1252 member of
- the Latin family and the probe decodes byte-identically under
- windows-1252, rewrite to windows-1252.
-* **MIN_EMIT_CONFIDENCE = 0.01** — noise candidates below this
- threshold are not emitted into the pool. Prevents JunkFilter from
- scoring weak coincidence picks against NB's confident top.
+* **Probes under 2 bytes** → windows-1252 at 0.1 confidence. The WHATWG default;
+ the detector never returns an empty result.
+* **Pure-ASCII probes** (no bytes ≥ 0x80, no nulls) → ISO-2022 structural
+ detection first, otherwise windows-1252. Bigram NB cannot discriminate Latin
+ code pages on pure-ASCII content, so return the HTML5-canonical answer.
+* **Latin-sibling rewrite** — on low-evidence probes (< 5 high bytes), if the top
+ NB candidate is a non-1252 member of the Latin family and the probe decodes
+ byte-identically under windows-1252, rewrite to windows-1252.
+* **Margin-gated emission** — at most `DEFAULT_TOP_K` (5) candidates are emitted,
+ and a runner-up only if its score is within
+ `MARGIN_THRESHOLD_NATS_PER_BIGRAM` (0.20) × scored bigrams of top-1.
+ Confidence is that linear margin distance, not a softmax: top-1 is 1.0, a
+ candidate at half the threshold is 0.5, and one at the threshold is 0.0 and is
+ dropped. So JunkFilter never scores a weak coincidence pick against NB's
+ confident top.
==== CJK decode-failure veto (`CjkDecodeValidator`)
A legacy multi-byte CJK class (GB18030, Big5-HKSCS, Shift_JIS, EUC-JP,
x-windows-949, x-EUC-TW) that NB picks on Latin/Cyrillic/garbage bytes is
-*false-CJK*: those bytes don't validate under the charset, so decoding
-produces many malformed/unmappable events, whereas real CJK decodes
-cleanly. After NB, each legacy-CJK candidate is decoded under its vendor
-superset (`CharsetSupersets`) and its failure rate measured as
-`failures / high-bytes`; above ~2.5% the candidate is dropped — and if it
-was NB's only pick, the pool empties and windows-1252 wins. Two
-corrections make the rate trustworthy:
-
-* **Decode under the vendor superset, not the strict base** — real
- vendor-extension chars (NEC/IBM for Shift_JIS/EUC-JP, HKSCS for Big5)
- would otherwise count as failures and penalize genuine CJK.
-* **Discount embedded UTF-8** — mixed-encoding pages (legacy CJK body +
- UTF-8 widgets) would otherwise read as 2–9.5% failure. The validator
- walks the bytes and *skips* positions that begin a valid UTF-8 sequence
- (it does NOT physically strip them — that would misalign a pure
- legacy-CJK stream and manufacture failures), decoding the legacy charset
- in place elsewhere. Post-discount, real CJK (pure or mixed) is ≤1.6%
- while genuine false-CJK stays ≥5.3%, so ~2.5% separates them.
-
-This veto catches *structurally-illegal* false-CJK only. The
-*legal-but-wrong* class — Latin/Cyrillic bytes that form a *valid* CJK
-decode at ~0 failure — is the typicality layer's job (<<junk-filter>>),
-not this veto's.
+*false-CJK*: those bytes do not validate under the charset, so decoding produces
+many malformed/unmappable events, whereas real CJK decodes cleanly. Each
+legacy-CJK candidate is decoded under its vendor superset (`CharsetSupersets`)
+and its `failures / high-bytes` rate measured; above ~2.5% the candidate is
+dropped, and if it was NB's only pick the pool empties and windows-1252 wins.
+
+Two corrections make the rate trustworthy:
+
+* **Decode under the vendor superset, not the strict base** — real vendor
+ extensions (NEC/IBM for Shift_JIS/EUC-JP, HKSCS for Big5) would otherwise count
+ as failures and penalize genuine CJK.
+* **Discount embedded UTF-8** — mixed-encoding pages (legacy CJK body + UTF-8
+ widgets) would otherwise read as 2–9.5% failure. The validator *skips*
+ positions that begin a valid UTF-8 sequence rather than physically stripping
+ them, which would misalign a pure legacy-CJK stream and manufacture failures.
+ Post-discount, real CJK is ≤1.6% and genuine false-CJK ≥5.3%, so ~2.5%
+ separates them.
+
+The veto catches *structurally illegal* false-CJK only. The *legal-but-wrong*
+class — Latin/Cyrillic bytes that form a valid CJK decode at ~0 failures — is the
+typicality layer's job (<<junk-filter>>).
[[junk-filter]]
== JunkFilterEncodingDetector — text-quality arbitration
`JunkFilterEncodingDetector` is a `MetaEncodingDetector` backed by
-`JunkDetector`, a script-aware text-quality scorer that distinguishes
-clean text from mojibake / wrong-codec / corrupted decodings. Its
-presence switches the composite into collect-all mode.
+`JunkDetector`, a script-aware text-quality scorer that distinguishes clean text
+from mojibake, wrong-codec, and corrupted decodings. Its presence switches the
+composite into collect-all mode.
[IMPORTANT]
====
-*JunkFilter arbitrates over ALL candidate tiers, including
-DECLARATIVE.* Real-world web declarations are unreliable, so a
-`<meta charset>` tag is one input among several.
+*JunkFilter arbitrates over ALL candidate tiers, including DECLARATIVE.*
+Real-world web declarations are unreliable, so a `<meta charset>` tag is one
+input among several.
-A declared charset wins when its decoding is byte-identical to at
-least one other candidate's — text quality can't distinguish the two,
-so honouring the declaration ensures downstream divergent bytes
-(e.g. 0xA4 = € in ISO-8859-15, ¤ in windows-1252) are decoded as the
-author intended.
+A declared charset wins when its decoding is byte-identical to at least one other
+candidate's: text quality cannot distinguish the two, so honouring the
+declaration ensures downstream divergent bytes (0xA4 = € in ISO-8859-15, ¤ in
+windows-1252) decode as the author intended.
-For plain first-match-wins, omit JunkFilter (see <<opting-out-of-arbitration>>).
+For plain first-match-wins, omit JunkFilter — see <<opting-out-of-arbitration>>.
====
=== Arbitration outline
. **One unique candidate** — abstain; composite default ordering wins.
-. **HTML byte-stripping** — `HtmlByteStripper` removes well-formed
- HTML/XML markup before decoding so the score reflects body text, not
- tags. Falls back to the raw probe if no tags are detected.
-. **Decode under each candidate** — insertion-ordered for deterministic
+. **HTML byte-stripping** — `HtmlByteStripper` removes markup before decoding, so
+ the score reflects body text. Falls back to the raw probe if no tags are found.
+. **Decode under each candidate** — insertion-ordered, for deterministic
tournament seeding.
-. **Declarative-equivalent-decode preference** — if a DECLARATIVE
- candidate's decoded output equals at least one other candidate's, it
- wins immediately.
+. **Declarative-equivalent-decode preference** — a DECLARATIVE candidate whose
+ decoded output equals another candidate's wins immediately.
. **All decodings identical, no DECLARATIVE** — abstain.
-. **Pairwise tournament** — first candidate seeds champion; each
- challenger compared via `JunkDetector.compare`; higher z-score wins.
+. **Pairwise tournament** — the first candidate seeds the champion; each
+ challenger is compared via `JunkDetector.compare`, higher score wins.
=== Post-tournament demote gates
-Two demote-only refinements run after the champion is chosen. Each fires only
-to *demote* the champion across one boundary the whole-text z-score reads
-poorly under COMMON-dilution; neither can promote, so they cannot cost a
-confident detection.
-
-* **CJK family gate** — the whole-text z coin-flips on the CJK/non-CJK boundary
- when markup and digits decode identically and swamp the few discriminating
- high bytes. A script-letter "diff" z — scored over only the `>= 0x80`
- letters/ideographs, where candidates actually differ — reads that boundary
- cleanly. If the champion is CJK and the best non-CJK diff-z beats the best
- CJK diff-z by `FAMILY_DIFF_MARGIN` (2.0), demote to the best non-CJK
- candidate. The reverse (promote to CJK) regressed at scale and is
- unnecessary — genuine CJK is `<meta>`-declared upstream.
-
-* **Within-Latin letter gate** — among single-byte Latin siblings the z also
- coin-flips, occasionally promoting a DOS-OEM / Mac charset (IBM850,
- x-MacRoman) whose high bytes decode to box-drawing / symbols over the
- windows-1252 truth. Cased-letter count reads this where typicality cannot:
- if the champion is a Latin SBCS, a windows-1252 candidate is present, the
- probe is high-byte-dense, and windows-1252 decodes clearly more cased
- high-byte letters (by a margin), demote to windows-1252. Directional — a
- genuine Central-European / DOS document has *more* letters under its true
- charset, so the gate stays silent. Latin-scoped, so it never crosses the
- CJK boundary (the family gate's job) or touches a non-Latin SBCS, whose
- Cyrillic/Greek cased letters would pollute the count. Shares the
- `HighByteLetterStats` letter counter with Mojibuster's Western-Latin sibling
- fallback.
+Two refinements run after the champion is chosen. Each fires only to *demote*
+across one boundary the whole-text score reads poorly under COMMON-dilution;
+neither can promote, so neither can cost a confident detection.
+
+* **CJK family gate** — the whole-text score coin-flips on the CJK/non-CJK
+ boundary when markup and digits decode identically and swamp the few
+ discriminating high bytes. A script-letter "diff" score, over only the
+ `>= 0x80` letters and ideographs where candidates actually differ, reads that
+ boundary cleanly. If the champion is CJK and the best non-CJK diff score beats
+ the best CJK diff score by `FAMILY_DIFF_MARGIN` (2.0), demote to the best
+ non-CJK candidate. The reverse direction is unnecessary — genuine CJK is
+ `<meta>`-declared upstream — and regressed at scale.
+
+* **Within-Latin letter gate** — among single-byte Latin siblings the score also
+ coin-flips, occasionally promoting a DOS-OEM or Mac charset (IBM850,
+ x-MacRoman) whose high bytes decode to box-drawing and symbols over the
+ windows-1252 truth. Cased-letter count reads this where typicality cannot: if
+ the champion is a Latin SBCS, a windows-1252 candidate is present, the probe is
+ high-byte-dense, and windows-1252 decodes clearly more cased high-byte letters,
+ demote to windows-1252. The gate is directional — a genuine Central-European
+ or DOS document has *more* letters under its true charset, so it stays silent —
+ and Latin-scoped, so it never crosses the CJK boundary or touches a non-Latin
+ SBCS whose Cyrillic/Greek cased letters would pollute the count. It shares
+ `HighByteLetterStats` with Mojibuster's Latin-sibling fallback.
=== JunkDetector scoring
-`JunkDetector` partitions decoded text into maximal Unicode-script runs
-(COMMON / INHERITED / UNKNOWN attach to the preceding run). Each run
-is scored against its own per-script model (byte-bigram log-prob table
-+ named-Unicode-block transitions + control-byte fraction) into a
-per-run logit. Logits are combined into an overall z-score weighted by
-run byte count, so mixed-script text is scored fairly.
+`JunkDetector` buckets every codepoint bigram to its Unicode script (COMMON,
+INHERITED and UNKNOWN glue folded into the adjacent script) and scores it against
+that script's codepoint-bigram table, calibrated per script. Global
+document-level features — block transitions, control-byte fraction, script
+transitions, replacement ratio, script alternation — are combined with the
+per-script term by a single linear combiner. Positive output means clean.
-Scripts not in the trained model are treated as neutral (z = 0), not
-junk — this prevents a garbled-but-recognisable decoding from beating
-a correct decoding whose script is unknown to the model.
+Scripts not in the trained model are treated as neutral (0), not junk, so a
+garbled-but-recognisable decoding cannot beat a correct decoding whose script the
+model does not know.
+
+See xref:advanced/junk-detection.adoc[Text Quality Scoring] for the scorer's own
+API and xref:advanced/junk-detection-build.adoc[Building the Junk Detector] for
+its model.
=== Why HTML stripping is essential
-The score is a byte-weighted average over script runs. An unstripped
-HTML probe with 10 KB of whitespace + tags and 24 bytes of body gives
-the body a weight of ~0.0024 — invisible. Stripping gets the probe to
-a content-only byte budget where scripts compete on comparable footing.
+The score is byte-weighted. An unstripped HTML probe with 10 KB of whitespace
+and tags plus 24 bytes of body gives the body a weight of ~0.0024 — invisible.
+Stripping gets the probe to a content-only byte budget where scripts compete on
+comparable footing.
-`HtmlByteStripper` operates on raw bytes (ASCII tag delimiters survive
-every charset Tika cares about). `<script>`, `<style>`, and comment
-bodies are dropped; text-bearing attribute values (`alt`, `title`,
-`placeholder`, `aria-label`, `summary`, `label`) are preserved.
+`HtmlByteStripper` operates on raw bytes, since ASCII tag delimiters survive
+every charset Tika cares about.
[[opting-out-of-arbitration]]
=== Opting out — first-match-wins
-This is *not* the default chain — it's a custom configuration.
-By default, JunkFilter arbitrates across all candidates as described
-above. To opt out, omit `JunkFilterEncodingDetector` from the chain;
-the composite detector then runs in plain first-match-wins mode (each
-base detector's top result is taken in registration order):
+This is *not* the default chain. Omitting `JunkFilterEncodingDetector` puts the
+composite detector in plain first-match-wins mode, taking each base detector's
+top result in registration order:
[source,json]
----
@@ -381,207 +384,125 @@ base detector's top result is taken in registration order):
}
----
-In this opt-out chain, BOM / metadata / meta-tag declarations are
-honoured authoritatively when present, and NB only runs when no
-declaration fires. Trade-off: lying declarations propagate
-unfiltered, and Mojibuster's top statistical guess is taken without
-any text-quality cross-check.
-
-== Performance
-
-Evaluated on a held-out 37-charset devtest corpus (~1.22M probes per
-probe length). Raw outputs are preserved in
+BOM, metadata, and meta-tag declarations are then authoritative when present, and
+NB runs only when no declaration fires. The trade-off: lying declarations
+propagate unfiltered, and Mojibuster's top statistical guess is taken with no
+text-quality cross-check.
+
+== Accuracy and latency
+
+Held-out MADLAD-400 + Wikipedia devtest, 1 469 647 samples across 41 charsets
+including the structural-only ones (US-ASCII, ISO-2022-JP/KR/CN, UTF-32-BE/LE).
+"Mojibuster (All)" is the production configuration: NB model + structural
+pre-filters + all post-processing rules. UTF-32 is handled by the structural
+pre-filter (a validity-only codepoint check, in the spirit of ICU4J's
+`CharsetRecog_UTF_32`) and UTF-16 by the structural phases plus the specialist
+model, so neither appears among the NB model's 34 classes.
+
+Four metrics are reported. **Strict** = exact charset name match. **Soft** =
+exact or confusable-group match (predicting IBM500 for an IBM1047 file is
+soft-correct; they share 247 of 256 byte mappings). **Decode-match** = the
+predicted charset decodes the probe to the same string as the true charset, so
+the prediction is functionally correct even when the label differs.
+**Alpha-match** = decode-match ignoring non-alphanumeric characters.
+
+IMPORTANT: No accuracy figures are published here. The last full run predates
+the `nb-bigram.bin` requantization in TIKA-4745, and no test asserts an accuracy
+floor, so nothing has confirmed the model's numbers against the shipped binary
+since. Run `EvalCharsetDetectors` to measure the current model.
+
+The qualitative shape of that run, which a requantization is not expected to
+change: Mojibuster roughly doubles ICU4J's strict accuracy and roughly triples
+juniversalchardet's, at an order of magnitude less latency than ICU4J.
+Decode-match runs well ahead of strict accuracy, because most residual errors
+pick a charset that decodes to the same string anyway. Accuracy climbs steeply
+with probe length and is weakest at 8 bytes — the default chain trades occasional
+overrides of correct declarations on short probes for resilience against lying
+declarations and wrong-codec decodings.
+
+Earlier ablation runs are preserved under
`tika-encoding-detectors/tika-encoding-detector-mojibuster/docs/performance/`.
-The default chain trades occasional overrides of correct declarations
-on short probes for resilience against lying declarations and wrong-codec
-decodings.
-
-== MetadataCharsetDetector
-
-A lightweight detector in `tika-core` that reads declarative charset hints from
-the `Metadata` object before any byte analysis:
-
-* `Content-Type` charset parameter (e.g. `text/html; charset=windows-1251`)
-* `Content-Encoding` (used by `RFC822Parser` and similar MIME-aware parsers)
-
-Applies WHATWG label normalization: `ISO-8859-1` and `US-ASCII` are mapped to
-`windows-1252` because browsers (and the HTML5 spec) treat them as aliases for
-windows-1252 in practice.
-
-Returns a DECLARATIVE result, which `JunkFilterEncodingDetector` honours when
-its decoding is byte-identical to at least one other candidate's.
-
-== BOMDetector
-
-Reads the first 4 bytes and detects:
-
-[cols="1,1", options="header"]
-|===
-| Byte sequence | Encoding
-
-| `EF BB BF` | UTF-8
-| `FF FE 00 00` | UTF-32-LE
-| `00 00 FE FF` | UTF-32-BE
-| `FF FE` | UTF-16-LE
-| `FE FF` | UTF-16-BE
-|===
-
-Returns a DECLARATIVE result. The HTML detectors do not handle BOMs on their
-own: `BOMDetector` is the sole source of BOM evidence, which lets
-`JunkFilterEncodingDetector` arbitrate when a BOM and a `<meta charset>` tag
-disagree.
+== Why Naive Bayes rather than hashed maxent
-== Performance and accuracy
+* **Speed** — direct bigram indexing removes the hash and bucket lookup. The
+ inner loop is `score[c] += logP[b × numClasses + c] × idf[b]` with no branching,
+ since zero-IDF bigrams are skipped before the class loop. Roughly an order of
+ magnitude faster than ICU4J, and modestly faster than juniversalchardet, with
+ better accuracy than both at every probe length.
+* **Memory layout** — bigram-major byte arrays keep the full table in L3 cache,
+ and the hot loop walks it sequentially.
+* **Interpretability** — per-class training totals, vocabulary sizes, and per-class
+ α are all visible in the saved model, which makes calibration problems (class
+ bias from training size, smoothing spread) diagnosable rather than mysterious.
-Numbers are from the held-out MADLAD-400 + Wikipedia devtest set (model v6).
-1 469 647 samples across 41 charsets including structural-only charsets
-(US-ASCII, ISO-2022-JP/KR/CN, UTF-32-BE/LE).
-
-"All" = ML model + structural pre-filters + all post-processing rules — this
-is the production configuration. UTF-32 is detected by a structural
-pre-filter (validity-only codepoint check inspired by ICU4J's
-`CharsetRecog_UTF_32`); UTF-16 uses a combination of structural phases
-(null-column, low-block-prefix) and the ML model for CJK cases. The ML model
-has 35 classes (UTF-32 labels are excluded from training since they are handled
-structurally).
-
-Raw eval output: `eval-v6-no-utf32.txt` in this directory.
-
-=== Overall accuracy
-
-[cols="2,1,1,1", options="header"]
-|===
-| Metric | Mojibuster (All) | ICU4J | juniversalchardet
-
-| Strict accuracy (full probe) | **95.0%** | 45.5% | 33.5%
-| Soft accuracy (full probe) | **97.3%** | 67.0% | 42.0%
-| Decode-match (full probe) | **99.4%** | 54.8% | 41.2%
-| Alpha-match (full probe) | **99.8%** | 55.3% | 43.5%
-| Latency (full probe) | **~13 µs** | ~147 µs | ~16 µs
-|===
-
-**Strict accuracy** = exact charset name match. +
-**Soft accuracy** = exact or confusable-group match (e.g. predicting IBM500 for
-an IBM1047 file counts as soft-correct since they share 247 of 256 byte
-mappings). +
-**Decode-match** = the predicted charset decodes the probe bytes to the same
-string as the true charset (the prediction is functionally correct even if the
-label differs). +
-**Alpha-match** = same as decode-match but ignoring non-alphanumeric characters.
-
-=== Accuracy by probe length
-
-[cols="1,1,1,1,1,1", options="header"]
-|===
-| Probe length | Strict% | Soft% | Top-3% | Decode% | Alpha%
-
-| 8 bytes | 59.1 | 62.6 | 70.2 | 83.2 | 83.4
-| 32 bytes | 80.8 | 83.6 | 86.3 | 93.4 | 93.5
-| 128 bytes | 91.4 | 93.8 | 94.2 | 97.4 | 97.5
-| full | 95.0 | 97.3 | 97.3 | 99.4 | 99.8
-|===
-
-=== Strict accuracy vs baseline detectors (by probe length)
-
-[cols="1,1,1,1", options="header"]
-|===
-| Probe length | NB pipeline | ICU4J | juniversalchardet
-
-| 20 bytes | **70.8%** | 20.5% | 22.1%
-| 100 bytes | **86.0%** | 37.4% | 27.8%
-| 256 bytes | **89.8%** | 40.5% | 28.2%
-| full probe | **91.6%** | 43.0% | 28.4%
-|===
+== On-disk model format
-=== Latency (µs per probe)
+`.bin` files use magic `NBB3`:
-[cols="1,1,1,1", options="header"]
-|===
-| Probe length | NB pipeline | ICU4J | juniversalchardet
-
-| 20 bytes | 2.5 | 31 | 8
-| 100 bytes | 5.8 | 86 | 14
-| 256 bytes | 7.9 | 136 | 19
-| full probe | 14.6 | 270 | 34
-|===
+[source]
+----
+int32 magic 0x4E424233 ("NBB3")
+int32 version (3)
+int32 numClasses
+float32 idfScale (global)
+byte[65536] idf8 (int8 quantized, non-negative)
+
+for each class:
+ uint16 labelLen, UTF-8 label bytes
+ float32 scale (per-class dequant)
+ byte unseenQ (int8 quantized unseen floor)
+ int32 vocabSize (number of trained pairs)
+ bigram keys ascending, each stored as:
+ varint deltaFromPrevKey (LEB128; first is the key itself)
+ byte logP8 (int8 quantized)
+----
-NB's inner loop is pure int8 multiply-accumulate with a direct-indexed
-bigram table; no hashing, no feature extraction pass. Approximately
-1 order of magnitude faster than ICU4J and 2-3× faster than
-juniversalchardet at every probe length, with better accuracy.
+Only trained pairs are stored. The loader materializes a dense
+`logP8[65 536 × numClasses]` array filled with per-class unseen floors and
+overwrites it with the trained values. The shipped 34-class model is ~680 KB on
+disk and ~2.2 MB resident.
== Training
-Training tools live in `org.apache.tika.ml.chardetect.tools`. Core
-flow:
+Tools live in `org.apache.tika.ml.chardetect.tools` (`tika-ml/tika-ml-chardetect`,
+packaged as a fat jar by the `train` profile).
[source,bash]
----
-# 1. Train (bigram Naive Bayes, per-class V99.9% vocab, int8 quantized)
+# 1. Train (bigram Naive Bayes, per-class 99.9% vocab, int8 quantized)
java -cp tika-ml/tika-ml-chardetect/target/tika-ml-chardetect-*-tools.jar \
org.apache.tika.ml.chardetect.tools.TrainNaiveBayesBigram \
- --data ~/datasets/charset-detect/train \
+ --data <corpus>/charset-detect/train \
--output nb-bigram.bin \
--coverage 0.999 \
--alpha-base 1.0 \
--max-samples-per-class 50000
-# 2. Install model into the mojibuster module resources
+# 2. Install the model into the mojibuster module resources
cp nb-bigram.bin \
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/\
org/apache/tika/ml/chardetect/nb-bigram.bin
# 3. Rebuild
-mvn install -pl tika-encoding-detectors/tika-encoding-detector-mojibuster \
+./mvnw clean install -pl tika-encoding-detectors/tika-encoding-detector-mojibuster \
-DskipTests
# 4. Evaluate against ICU4J and juniversalchardet on devtest
java -cp tika-ml/tika-ml-chardetect/target/tika-ml-chardetect-*-tools.jar \
org.apache.tika.ml.chardetect.tools.EvalCharsetDetectors \
--nb-model nb-bigram.bin \
- --data ~/datasets/charset-detect/devtest \
- --lengths 20,100,256,full
+ --data <corpus>/charset-detect/devtest \
+ --lengths 8,32,128,full
----
-Training data is sourced from MADLAD-400 (multilingual web text) and
-Wikipedia dumps, encoded to each target charset with charset-specific
-normalization (Arabic Yeh forms for windows-1256, Hebrew nikkud
-stripping for IBM424, punctuation mapping for legacy charsets that
-can't encode typographic characters).
-
-== On-disk model format
-
-`.bin` files use magic `NBB3` and store:
-
-* IDF scale + quantized IDF table (65 536 bytes).
-* Per class: label, dequantization scale, quantized unseen-floor
- value, vocabulary size, and each trained bigram as
- `(uint16 bigram, int8 logP)` pairs.
-
-Files for the shipped 34-class model are ~1 MB on disk. Loader
-materializes a dense `logP8[65 536 × numClasses]` array filled with
-per-class unseen floors, overwritten by trained pairs. Working-set
-memory: ~2 MB.
-
-== Why Naive Bayes
-
-Earlier Tika versions used multinomial logistic regression (maxent)
-with feature hashing. The move to NB was driven by:
-
-* **Speed**: direct bigram indexing removes the hash + bucket-lookup
- cost. Inner loop is `score[c] += logP[b × numClasses + c] × idf[b]`
- with no branching (zero-IDF bigrams are skipped before the class
- loop). Measured ~15 µs on a full 1 KB probe for 34 classes.
-* **Memory layout**: bigram-major byte arrays fit in L3 cache for the
- full table. Sequential access through the hot loop is cache-line
- efficient.
-* **Interpretability**: per-class training totals, vocabulary sizes,
- and per-class α are all visible in the saved model. Makes
- calibration issues (class bias from training size, smoothing spread,
- etc.) diagnosable.
+Training data comes from MADLAD-400 (multilingual web text) and Wikipedia dumps,
+encoded to each target charset with charset-specific normalization: Arabic Yeh
+forms for windows-1256, Hebrew nikkud stripping for IBM424, punctuation mapping
+for legacy charsets that cannot encode typographic characters.
== See also
-* Configuration: xref:../configuration/encoding-detectors.adoc[Configuring Encoding Detectors]
-* Language detection: xref:advanced/language-detection.adoc[Language Detection]
+* xref:configuration/encoding-detectors.adoc[Configuring Encoding Detectors]
+* xref:advanced/junk-detection.adoc[Text Quality Scoring (Junk Detection)]
+* xref:advanced/language-detection.adoc[Language Detection]
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-eval-20260417.txt b/docs/modules/ROOT/pages/advanced/charset-detection-eval-20260417.txt
deleted file mode 100644
index df584be3e3..0000000000
--- a/docs/modules/ROOT/pages/advanced/charset-detection-eval-20260417.txt
+++ /dev/null
@@ -1,240 +0,0 @@
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-Model: chardetect.bin (28 classes, post-stride-2 retrain, 2026-04-17)
-Devtest corpus: 33 classes = 28 TODAY_SBCS_INCLUDE + EUC-KR (CharsetSupersets test) + UTF-16-LE/BE + UTF-32-LE/BE
-Columns: Stat=model only | +ISO=+STRUCTURAL_GATES+C1-correction | +CJK=+grammar | All=ML+rules
-Metrics: R%=strict S%=soft T3%=top-3 D%=decode-match A%=alpha-match
-Baselines: ICU4J, juniversalchardet
-Note: results are Mojibuster ablations only — no CharSoup arbitration (see charset-20260417-plan.md TODO).
-
-=== Probe length: 20B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 79.7 79.7 80.5 83.8 83.8 | 80.6 80.6 81.6 84.7 84.7 | 80.6 80.6 81.6 84.7 84.7 | 80.6 80.6 81.6 84.7 84.7 | 0.0 14.5 71.8 16.8 17.1 | 0.0 44.7 44.7 47.2 47.5 |
-EUC-JP 37043 | 79.9 79.9 81.2 87.2 87.6 | 79.6 79.6 80.9 86.9 87.3 | 79.6 79.6 80.9 86.9 87.3 | 79.6 79.6 80.9 86.9 87.3 | 0.0 0.0 14.8 6.8 7.8 | 64.6 64.6 64.6 71.9 72.9 |
-EUC-KR 36883 | 0.0 89.8 90.2 89.8 89.9 | 0.0 87.4 87.8 90.3 90.4 | 0.0 87.4 87.8 90.3 90.4 | 0.0 87.4 87.8 90.3 90.4 | 0.0 0.0 28.6 2.8 3.0 | 83.9 83.9 83.9 86.7 87.0 |
-GB18030 36862 | 76.4 76.4 77.1 82.4 82.6 | 76.8 76.8 77.5 83.0 83.1 | 76.8 76.8 77.5 83.0 83.1 | 76.8 76.8 77.5 83.0 83.1 | 0.1 0.1 7.8 6.0 6.6 | 46.1 46.1 46.1 52.1 52.8 |
-IBM500 31455 | 95.8 95.8 95.8 95.8 95.8 | 62.0 62.0 62.0 62.0 62.0 | 62.0 62.0 62.0 62.0 62.0 | 62.0 62.0 62.0 62.0 62.0 | 92.1 92.1 98.3 92.1 92.1 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 30539 | 36.0 36.0 41.9 96.2 96.3 | 36.2 36.2 41.9 96.4 96.4 | 36.2 36.2 41.9 96.4 96.4 | 36.2 36.2 41.9 96.4 96.4 | 0.0 0.0 0.0 56.5 57.3 | 0.0 0.0 0.0 57.4 58.3 |
-IBM852 35403 | 50.4 50.4 58.3 96.8 96.8 | 50.5 50.5 58.0 96.8 96.8 | 50.5 50.5 58.0 96.8 96.8 | 50.5 50.5 58.0 96.8 96.8 | 0.0 0.0 0.0 39.2 39.4 | 0.0 0.0 0.0 40.8 41.1 |
-IBM855 36702 | 89.5 89.5 89.7 91.3 91.3 | 89.6 89.6 89.9 91.4 91.4 | 89.6 89.6 89.9 91.4 91.4 | 89.6 89.6 89.9 91.4 91.4 | 0.0 0.0 0.0 1.7 1.7 | 93.1 93.1 93.1 94.8 94.9 |
-IBM866 36985 | 93.4 93.4 93.9 95.6 95.6 | 94.1 94.1 94.6 96.3 96.3 | 94.1 94.1 94.6 96.3 96.3 | 94.1 94.1 94.6 96.3 96.3 | 52.3 52.3 79.1 54.4 54.4 | 94.9 94.9 94.9 97.0 97.0 |
-ISO-8859-16 32899 | 50.1 50.1 99.8 99.0 99.0 | 48.9 48.9 49.7 97.7 97.7 | 48.9 48.9 49.7 97.7 97.7 | 17.6 17.6 17.8 97.7 97.7 | 0.0 0.0 0.0 84.2 84.6 | 0.0 0.0 0.0 82.1 82.8 |
-ISO-8859-3 35648 | 46.3 46.3 47.7 98.7 98.7 | 46.0 46.0 47.2 98.4 98.4 | 46.0 46.0 47.2 98.4 98.4 | 45.3 45.3 46.5 98.4 98.4 | 0.0 0.0 0.0 50.9 50.9 | 0.0 0.0 0.0 53.2 53.2 |
-KOI8-R 36850 | 79.0 93.5 93.8 95.7 95.7 | 79.0 93.5 93.8 95.7 95.7 | 79.0 93.5 93.8 95.7 95.7 | 79.0 93.5 93.8 95.7 95.7 | 66.6 66.6 77.7 68.6 68.6 | 96.1 96.1 96.1 98.3 98.3 |
-KOI8-U 36846 | 84.2 96.3 96.4 96.8 96.8 | 84.2 96.2 96.4 96.8 96.8 | 84.2 96.2 96.4 96.8 96.8 | 84.2 96.2 96.4 96.8 96.8 | 0.0 59.5 73.2 19.3 19.4 | 0.0 97.1 97.1 33.2 33.2 |
-Shift_JIS 36917 | 83.4 83.4 84.7 90.6 90.8 | 79.4 79.4 81.3 86.6 86.7 | 79.4 79.4 81.3 86.6 86.7 | 79.4 79.4 81.3 86.6 86.7 | 0.0 0.0 2.9 6.9 7.3 | 67.2 67.2 67.2 74.4 75.0 |
-UTF-16-BE 36799 | 0.0 0.0 0.0 0.0 0.0 | 92.2 92.2 92.2 92.2 92.2 | 92.2 92.2 92.2 92.2 92.2 | 92.2 92.2 92.2 92.2 92.2 | 36.5 36.5 38.6 36.5 36.5 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 0.0 0.0 0.0 0.0 0.0 | 92.4 92.4 92.4 92.4 92.4 | 92.4 92.4 92.4 92.4 92.4 | 92.4 92.4 92.4 92.4 92.4 | 36.8 36.8 39.0 36.8 36.8 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 75.3 75.3 93.3 92.4 92.4 | 78.9 78.9 78.9 96.1 96.1 | 78.9 78.9 78.9 96.1 96.1 | 78.9 78.9 78.9 96.1 96.1 | 82.0 82.0 91.4 97.9 97.9 | 81.9 81.9 81.9 99.0 99.0 |
-windows-1250 34499 | 33.9 33.9 51.1 89.2 89.5 | 33.9 33.9 50.7 88.7 89.0 | 33.9 33.9 50.7 88.7 89.0 | 20.7 20.7 34.3 88.7 89.0 | 11.2 54.0 88.8 83.5 83.5 | 0.0 0.0 0.0 58.5 62.2 |
-windows-1251 36852 | 93.0 93.0 93.2 94.9 94.9 | 93.0 93.0 93.2 94.9 94.9 | 93.0 93.0 93.2 94.9 94.9 | 93.0 93.0 93.2 94.9 94.9 | 60.2 60.4 76.0 62.2 62.2 | 74.4 74.5 74.5 76.3 76.5 |
-windows-1252 25874 | 22.1 22.1 29.3 88.3 88.4 | 73.8 73.8 80.9 87.9 88.0 | 73.8 73.8 81.0 87.9 88.0 | 87.9 87.9 94.7 87.9 88.0 | 3.8 65.8 94.7 88.4 88.4 | 0.0 98.6 98.6 92.8 98.6 |
-windows-1253 36845 | 87.4 87.4 87.7 90.7 90.8 | 87.4 87.4 87.7 90.8 90.8 | 87.4 87.4 87.7 90.8 90.8 | 87.4 87.4 87.7 90.8 90.8 | 2.0 72.2 87.0 74.7 74.8 | 0.1 89.9 89.9 89.6 92.4 |
-windows-1254 36705 | 60.8 60.8 70.4 86.9 87.0 | 60.8 60.8 70.3 86.7 86.8 | 60.8 60.8 70.3 86.7 86.8 | 50.8 50.8 57.0 86.7 86.8 | 5.3 58.2 84.0 79.6 79.6 | 0.0 0.0 0.0 39.2 43.5 |
-windows-1255 31252 | 89.8 89.8 90.1 91.3 91.3 | 89.8 89.8 90.1 91.3 91.3 | 89.8 89.8 90.1 91.3 91.3 | 89.8 89.8 90.1 91.3 91.3 | 6.7 34.0 48.1 34.5 35.5 | 93.9 95.6 95.6 96.6 97.2 |
-windows-1256 41912 | 94.0 94.0 94.2 95.4 95.4 | 94.0 94.0 94.2 95.4 95.4 | 94.0 94.0 94.2 95.4 95.4 | 94.0 94.0 94.2 95.4 95.4 | 36.0 59.9 79.6 37.4 37.4 | 0.0 0.0 0.0 1.3 1.4 |
-windows-1257 30789 | 30.8 30.8 50.6 67.5 67.6 | 30.9 30.9 50.5 67.4 67.4 | 30.9 30.9 50.5 67.4 67.4 | 23.4 23.4 38.8 67.4 67.4 | 0.0 0.0 0.0 47.3 47.4 | 0.0 0.0 0.0 43.5 49.3 |
-windows-1258 36885 | 80.1 80.1 87.5 85.7 85.7 | 80.1 80.1 87.5 85.7 86.1 | 80.1 80.1 87.5 85.7 86.1 | 80.1 80.1 87.5 85.7 86.1 | 0.0 0.0 0.0 5.5 5.6 | 0.0 0.0 0.0 5.3 5.7 |
-windows-874 31440 | 76.8 76.8 78.0 85.8 85.8 | 78.2 78.2 79.6 87.2 87.2 | 78.2 78.2 79.7 87.2 87.2 | 78.2 78.2 79.7 87.2 87.2 | 0.0 0.0 0.0 8.9 8.9 | 0.0 0.0 0.0 83.8 94.6 |
-x-EUC-TW 26788 | 87.9 87.9 88.3 92.1 92.2 | 88.0 88.0 88.3 92.3 92.3 | 87.9 87.9 88.3 92.3 92.3 | 87.9 87.9 88.3 92.3 92.3 | 0.0 0.0 0.0 4.2 4.4 | 43.9 43.9 43.9 48.2 48.5 |
-x-MacRoman 1994 | 15.3 15.3 30.1 70.7 70.8 | 15.6 15.6 30.0 71.0 71.6 | 15.6 15.6 30.0 71.0 71.6 | 15.6 15.6 30.0 71.0 71.6 | 0.0 0.0 0.0 54.6 55.3 | 0.0 0.0 0.0 55.5 56.2 |
-x-mac-cyrillic 1773 | 45.4 45.4 45.6 45.4 45.4 | 45.5 45.5 45.7 45.5 45.5 | 45.5 45.5 45.7 45.5 45.5 | 45.5 45.5 45.7 45.5 45.5 | 0.0 0.0 0.0 0.0 0.0 | 51.0 51.0 51.0 51.0 51.0 |
-x-windows-949 36719 | 89.9 89.9 90.2 89.9 90.0 | 87.7 87.7 88.0 90.3 90.4 | 87.7 87.7 88.0 90.3 90.4 | 87.7 87.7 88.0 90.3 90.4 | 0.0 0.0 29.0 2.5 2.7 | 0.0 84.2 84.2 86.7 87.0 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1085250 | 60.3 64.2 69.0 78.4 78.5 | 73.5 77.4 80.1 90.6 90.6 | 73.5 77.4 80.1 90.6 90.6 | 71.9 75.8 78.1 90.6 90.6 | 23.1 34.2 45.9 45.1 45.2 | 27.7 40.5 40.5 54.5 55.7 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 20.5 | 15.4 | 15.0 | 15.1 | 23.5 | 6.7 |
-
-=== Probe length: 50B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 96.2 96.2 96.3 97.6 97.7 | 96.4 96.4 96.5 97.8 97.9 | 96.4 96.4 96.5 97.8 97.9 | 96.4 96.4 96.5 97.8 97.9 | 0.0 92.2 95.0 69.8 69.9 | 0.0 79.9 79.9 70.2 70.3 |
-EUC-JP 37043 | 95.2 95.2 95.4 97.1 97.2 | 95.1 95.1 95.3 97.1 97.1 | 95.1 95.1 95.3 97.1 97.1 | 95.1 95.1 95.3 97.1 97.1 | 85.1 85.1 87.2 87.0 87.3 | 89.3 89.3 89.3 91.2 91.4 |
-EUC-KR 36883 | 0.0 96.0 96.1 96.0 96.0 | 0.0 95.5 95.6 96.1 96.1 | 0.0 95.5 95.6 96.1 96.1 | 0.0 95.5 95.6 96.1 96.1 | 93.8 93.8 94.7 94.3 94.4 | 97.5 97.5 97.5 98.1 98.2 |
-GB18030 36862 | 93.8 93.8 93.9 95.7 95.8 | 93.9 93.9 94.1 95.9 95.9 | 93.9 93.9 94.1 95.9 95.9 | 93.9 93.9 94.1 95.9 95.9 | 81.1 81.1 85.6 83.0 83.3 | 87.3 87.3 87.3 89.2 89.5 |
-IBM500 31455 | 99.7 99.7 99.7 99.7 99.7 | 69.9 69.9 69.9 69.9 69.9 | 69.9 69.9 69.9 69.9 69.9 | 69.9 69.9 69.9 69.9 69.9 | 85.4 85.4 99.8 85.4 85.4 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 30539 | 64.3 64.3 68.4 97.1 97.1 | 64.4 64.4 68.4 97.2 97.2 | 64.4 64.4 68.4 97.2 97.2 | 64.4 64.4 68.4 97.2 97.2 | 0.0 0.0 0.0 30.7 31.4 | 0.0 0.0 0.0 30.8 31.5 |
-IBM852 35403 | 78.1 78.1 84.0 97.1 97.1 | 78.1 78.1 83.9 97.1 97.1 | 78.1 78.1 83.9 97.1 97.1 | 78.1 78.1 83.9 97.1 97.1 | 0.0 0.0 0.0 15.1 15.3 | 0.0 0.0 0.0 15.3 15.6 |
-IBM855 36702 | 99.0 99.0 99.0 99.3 99.3 | 99.0 99.0 99.0 99.3 99.3 | 99.0 99.0 99.0 99.3 99.3 | 99.0 99.0 99.0 99.3 99.3 | 0.0 0.0 0.0 0.3 0.3 | 98.9 98.9 98.9 99.1 99.2 |
-IBM866 36985 | 99.2 99.2 99.3 99.5 99.5 | 99.3 99.3 99.3 99.6 99.6 | 99.3 99.3 99.3 99.6 99.6 | 99.3 99.3 99.3 99.6 99.6 | 75.5 75.5 94.7 75.8 75.8 | 99.0 99.0 99.0 99.3 99.3 |
-ISO-8859-16 32899 | 76.0 76.0 99.8 99.3 99.3 | 75.7 75.7 76.2 99.0 99.0 | 75.7 75.7 76.2 99.0 99.0 | 42.1 42.1 42.3 99.0 99.0 | 0.0 0.0 0.0 76.1 76.4 | 0.0 0.0 0.0 68.0 68.5 |
-ISO-8859-3 35648 | 79.0 79.0 80.0 99.2 99.2 | 78.9 78.9 79.8 99.0 99.0 | 78.9 78.9 79.8 99.0 99.0 | 78.1 78.1 78.9 99.0 99.0 | 0.0 0.0 0.0 20.9 20.9 | 0.0 0.0 0.0 21.1 21.1 |
-KOI8-R 36850 | 93.9 99.3 99.3 99.6 99.6 | 93.9 99.3 99.3 99.6 99.6 | 93.9 99.3 99.3 99.6 99.6 | 93.9 99.3 99.3 99.6 99.6 | 86.4 86.4 94.8 86.7 86.7 | 99.2 99.2 99.2 99.5 99.5 |
-KOI8-U 36846 | 96.6 99.6 99.6 99.3 99.3 | 96.6 99.6 99.6 99.3 99.3 | 96.6 99.6 99.6 99.3 99.3 | 96.6 99.6 99.6 99.3 99.3 | 0.0 79.4 91.7 5.1 5.1 | 0.0 99.1 99.1 7.1 7.1 |
-Shift_JIS 36917 | 95.9 95.9 96.1 97.8 97.8 | 92.9 92.9 93.1 94.7 94.7 | 92.9 92.9 93.1 94.7 94.7 | 92.9 92.9 93.1 94.7 94.7 | 86.9 86.9 87.1 88.7 88.8 | 93.9 93.9 93.9 95.8 95.9 |
-UTF-16-BE 36799 | 0.0 0.0 0.0 0.0 0.0 | 96.4 96.4 96.4 96.4 96.4 | 96.4 96.4 96.4 96.4 96.4 | 96.4 96.4 96.4 96.4 96.4 | 69.8 69.8 93.2 69.8 69.8 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 0.0 0.0 0.0 0.0 0.0 | 96.5 96.5 96.5 96.5 96.5 | 96.5 96.5 96.5 96.5 96.5 | 96.5 96.5 96.5 96.5 96.5 | 70.2 70.2 93.3 70.2 70.2 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 85.7 85.7 94.8 93.8 93.8 | 91.6 91.6 91.6 99.7 99.7 | 91.6 91.6 91.6 99.7 99.7 | 91.6 91.6 91.6 99.7 99.7 | 90.4 90.4 95.0 98.4 98.4 | 91.3 91.3 91.3 99.4 99.4 |
-windows-1250 34499 | 62.6 62.6 76.8 91.5 92.0 | 62.5 62.5 76.7 91.3 91.9 | 62.6 62.6 76.7 91.3 91.9 | 48.6 48.6 60.5 91.3 91.9 | 25.5 64.7 96.4 81.8 81.8 | 0.0 0.0 0.0 31.9 35.8 |
-windows-1251 36852 | 98.9 98.9 99.0 99.3 99.3 | 98.9 98.9 99.0 99.3 99.3 | 98.9 98.9 99.0 99.3 99.3 | 98.9 98.9 99.0 99.3 99.3 | 82.1 82.1 90.8 82.5 82.5 | 83.7 83.7 83.7 84.1 84.1 |
-windows-1252 25874 | 43.2 43.2 50.4 89.1 89.2 | 75.2 75.2 82.4 89.0 89.1 | 75.2 75.2 82.4 89.0 89.1 | 88.9 88.9 95.7 89.0 89.1 | 8.2 80.5 98.4 91.1 91.1 | 0.0 98.8 98.8 89.0 98.8 |
-windows-1253 36845 | 96.6 96.6 96.6 97.1 97.1 | 96.6 96.6 96.6 97.1 97.1 | 96.6 96.6 96.6 97.1 97.1 | 96.6 96.6 96.6 97.1 97.1 | 5.3 92.7 96.9 91.7 91.7 | 0.2 96.3 96.3 89.7 95.3 |
-windows-1254 36705 | 90.7 90.7 94.8 94.9 94.9 | 90.7 90.7 94.8 94.9 94.9 | 90.7 90.7 94.8 94.9 94.9 | 86.0 86.0 88.6 94.9 94.9 | 13.5 80.8 97.0 85.4 85.4 | 0.0 0.0 0.0 10.2 12.3 |
-windows-1255 31252 | 98.3 98.3 98.3 98.7 98.7 | 98.3 98.3 98.3 98.7 98.7 | 98.3 98.3 98.3 98.7 98.7 | 98.3 98.3 98.3 98.7 98.7 | 16.1 52.8 63.7 50.9 53.2 | 97.8 98.7 98.7 98.7 99.1 |
-windows-1256 41912 | 98.1 98.1 98.1 98.5 98.5 | 98.1 98.1 98.1 98.5 98.5 | 98.1 98.1 98.1 98.5 98.5 | 98.1 98.1 98.1 98.5 98.5 | 42.4 71.9 90.8 42.8 42.8 | 0.0 0.0 0.0 0.4 0.4 |
-windows-1257 30789 | 62.5 62.5 80.1 74.9 74.9 | 62.5 62.5 80.0 74.9 74.9 | 62.5 62.5 80.0 74.9 74.9 | 52.7 52.7 65.1 74.9 74.9 | 0.0 0.0 0.0 28.2 28.2 | 0.0 0.0 0.0 25.6 30.3 |
-windows-1258 36885 | 98.0 98.0 98.6 98.7 98.7 | 98.0 98.0 98.6 98.7 98.8 | 98.0 98.0 98.6 98.7 98.8 | 98.0 98.0 98.6 98.7 98.8 | 0.0 0.0 0.0 0.8 0.8 | 0.0 0.0 0.0 0.7 0.8 |
-windows-874 31440 | 94.2 94.2 94.4 97.1 97.1 | 94.5 94.5 94.7 97.3 97.3 | 94.5 94.5 94.7 97.3 97.3 | 94.5 94.5 94.7 97.3 97.3 | 0.0 0.0 0.0 3.0 3.0 | 0.0 0.0 0.0 80.0 98.9 |
-x-EUC-TW 26788 | 97.1 97.1 97.1 98.5 98.6 | 97.1 97.1 97.1 98.6 98.6 | 97.1 97.1 97.1 98.6 98.6 | 97.1 97.1 97.1 98.6 98.6 | 0.0 0.0 0.0 1.5 1.5 | 77.1 77.1 77.1 78.6 78.6 |
-x-MacRoman 1994 | 34.9 34.9 57.1 62.2 62.2 | 35.8 35.8 57.5 63.1 63.1 | 35.8 35.8 57.5 63.1 63.1 | 35.8 35.8 57.5 63.1 63.1 | 0.0 0.0 0.0 27.3 27.6 | 0.0 0.0 0.0 27.3 27.7 |
-x-mac-cyrillic 1773 | 58.9 58.9 59.0 58.9 58.9 | 58.9 58.9 59.0 58.9 58.9 | 58.9 58.9 59.0 58.9 58.9 | 58.9 58.9 59.0 58.9 58.9 | 0.0 0.0 0.0 0.0 0.0 | 66.9 66.9 66.9 66.9 66.9 |
-x-windows-949 36719 | 96.1 96.1 96.1 96.1 96.2 | 95.6 95.6 95.6 96.2 96.2 | 95.6 95.6 95.6 96.2 96.2 | 95.6 95.6 95.6 96.2 96.2 | 0.0 93.9 94.8 94.3 94.4 | 0.0 97.5 97.5 98.0 98.1 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1085250 | 73.6 77.1 79.8 83.3 83.4 | 86.9 90.4 92.1 95.9 95.9 | 86.9 90.4 92.1 95.9 95.9 | 85.3 88.8 90.2 95.9 95.9 | 40.9 59.7 67.1 62.2 62.3 | 33.3 47.9 47.9 53.2 54.6 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 25.5 | 21.5 | 21.0 | 21.0 | 42.4 | 8.6 |
-
-=== Probe length: 100B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 99.6 99.6 99.6 99.7 99.7 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 0.0 99.2 99.4 62.6 62.6 | 0.0 84.3 84.3 62.4 62.4 |
-EUC-JP 37043 | 98.8 98.8 98.8 99.3 99.3 | 98.8 98.8 98.8 99.3 99.3 | 98.8 98.8 98.8 99.3 99.3 | 98.8 98.8 98.8 99.3 99.3 | 97.5 97.5 97.9 98.0 98.0 | 96.4 96.4 96.4 96.8 96.9 |
-EUC-KR 36883 | 0.0 99.0 99.0 99.0 99.0 | 0.0 98.9 98.9 99.0 99.0 | 0.0 98.9 98.9 99.0 99.0 | 0.0 98.9 98.9 99.0 99.0 | 99.4 99.4 99.5 99.5 99.5 | 99.6 99.6 99.6 99.7 99.7 |
-GB18030 36862 | 98.2 98.2 98.3 98.9 98.9 | 98.3 98.3 98.3 99.0 99.0 | 98.3 98.3 98.3 99.0 99.0 | 98.3 98.3 98.3 99.0 99.0 | 95.5 95.5 97.3 96.2 96.3 | 97.4 97.4 97.4 98.0 98.2 |
-IBM500 31455 | 100.0 100.0 100.0 100.0 100.0 | 83.5 83.5 83.5 83.5 83.5 | 83.5 83.5 83.5 83.5 83.5 | 83.5 83.5 83.5 83.5 83.5 | 88.3 88.3 99.9 88.3 88.3 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 30539 | 82.5 82.5 86.4 96.6 96.7 | 82.7 82.7 86.5 96.9 96.9 | 82.7 82.7 86.6 96.9 97.0 | 82.7 82.7 86.6 96.9 97.0 | 0.0 0.0 0.0 12.3 12.7 | 0.0 0.0 0.0 12.3 12.7 |
-IBM852 35403 | 91.5 91.5 94.4 97.6 97.6 | 91.5 91.5 94.5 97.6 97.6 | 91.6 91.6 94.6 97.7 97.7 | 91.6 91.6 94.6 97.7 97.7 | 0.0 0.0 0.0 4.5 4.6 | 0.0 0.0 0.0 4.5 4.6 |
-IBM855 36702 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.1 0.1 | 99.8 99.8 99.8 99.8 99.8 |
-IBM866 36985 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 91.2 91.2 99.0 91.3 91.3 | 99.7 99.7 99.7 99.8 99.8 |
-ISO-8859-16 32899 | 88.0 88.0 99.6 99.0 99.1 | 87.9 87.9 88.7 98.9 99.0 | 88.1 88.1 88.7 99.0 99.1 | 75.2 75.2 75.4 99.0 99.1 | 0.0 0.0 0.0 63.1 63.4 | 0.0 0.0 0.0 54.5 54.9 |
-ISO-8859-3 35648 | 93.6 93.6 95.3 98.1 98.1 | 93.6 93.6 95.3 98.1 98.1 | 93.6 93.6 95.3 98.1 98.2 | 93.6 93.6 95.0 98.1 98.2 | 0.0 0.0 0.0 5.1 5.1 | 0.0 0.0 0.0 5.0 5.0 |
-KOI8-R 36850 | 98.7 99.9 99.9 100.0 100.0 | 98.7 99.9 99.9 100.0 100.0 | 98.7 99.9 99.9 100.0 100.0 | 98.7 99.9 99.9 100.0 100.0 | 95.9 95.9 99.1 96.0 96.0 | 99.7 99.7 99.7 99.8 99.8 |
-KOI8-U 36846 | 99.4 99.9 99.9 99.8 99.8 | 99.4 99.9 99.9 99.8 99.8 | 99.4 99.9 99.9 99.8 99.8 | 99.4 99.9 99.9 99.8 99.8 | 0.0 92.0 98.0 0.7 0.7 | 0.0 99.6 99.6 0.9 0.9 |
-Shift_JIS 36917 | 99.1 99.1 99.2 99.6 99.6 | 96.7 96.7 96.8 97.2 97.2 | 96.7 96.7 96.8 97.2 97.2 | 96.7 96.7 96.8 97.2 97.2 | 98.0 98.0 98.1 98.5 98.5 | 98.6 98.6 98.6 99.1 99.2 |
-UTF-16-BE 36799 | 0.0 0.0 0.0 0.0 0.0 | 96.3 96.3 96.3 96.3 96.3 | 96.3 96.3 96.3 96.3 96.3 | 96.3 96.3 96.3 96.3 96.3 | 68.8 68.8 94.1 68.8 68.8 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 0.0 0.0 0.0 0.0 0.0 | 96.4 96.4 96.4 96.4 96.4 | 96.4 96.4 96.4 96.4 96.4 | 96.4 96.4 96.4 96.4 96.4 | 69.6 69.6 94.1 69.6 69.6 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 93.1 93.1 97.4 96.7 96.7 | 96.5 96.5 96.5 100.0 100.0 | 96.5 96.5 96.5 100.0 100.0 | 96.5 96.5 96.5 100.0 100.0 | 95.5 95.5 97.3 99.0 99.0 | 95.8 95.8 95.8 99.3 99.3 |
-windows-1250 34499 | 80.9 80.9 87.3 94.4 94.9 | 81.1 81.1 87.4 94.6 95.1 | 81.1 81.1 87.4 94.6 95.1 | 74.1 74.1 79.6 94.6 95.1 | 43.1 72.0 98.5 80.5 80.5 | 0.0 0.0 0.0 14.8 18.0 |
-windows-1251 36852 | 99.8 99.8 99.8 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 92.2 92.2 96.6 92.3 92.3 | 87.3 87.3 87.3 87.4 87.5 |
-windows-1252 25874 | 62.2 62.2 67.1 90.0 90.2 | 80.0 80.0 84.9 90.0 90.2 | 80.0 80.0 84.9 90.1 90.2 | 89.8 89.8 94.7 90.1 90.2 | 13.6 86.5 99.1 93.1 93.1 | 0.0 99.1 99.1 83.9 99.0 |
-windows-1253 36845 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 9.8 98.3 99.4 95.8 95.9 | 0.3 98.5 98.5 86.4 96.0 |
-windows-1254 36705 | 99.0 99.0 99.4 99.4 99.4 | 99.0 99.0 99.4 99.4 99.4 | 99.0 99.0 99.4 99.4 99.4 | 98.2 98.2 98.4 99.4 99.4 | 24.4 93.8 99.6 94.4 94.4 | 0.0 0.0 0.0 1.4 1.8 |
-windows-1255 31252 | 99.6 99.6 99.6 99.7 99.7 | 99.6 99.6 99.6 99.7 99.7 | 99.6 99.6 99.6 99.7 99.7 | 99.6 99.6 99.6 99.7 99.7 | 26.7 63.6 77.7 59.6 63.8 | 99.2 99.4 99.4 99.4 99.6 |
-windows-1256 41912 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 99.2 99.2 99.2 99.3 99.3 | 46.9 80.1 96.4 47.1 47.1 | 0.0 0.0 0.0 0.1 0.1 |
-windows-1257 30789 | 85.3 85.3 94.9 88.8 88.8 | 85.3 85.3 94.9 88.8 88.8 | 85.3 85.3 94.9 88.8 88.8 | 77.3 77.3 82.5 88.8 88.8 | 0.0 0.0 0.0 17.8 17.8 | 0.0 0.0 0.0 16.0 19.7 |
-windows-1258 36885 | 99.7 99.7 99.8 99.9 99.9 | 99.7 99.7 99.8 99.9 99.9 | 99.7 99.7 99.8 99.9 99.9 | 99.7 99.7 99.8 99.9 99.9 | 0.0 0.0 0.0 0.2 0.2 | 0.0 0.0 0.0 0.2 0.2 |
-windows-874 31440 | 98.8 98.8 98.8 99.5 99.5 | 98.8 98.8 98.9 99.5 99.5 | 98.8 98.8 98.9 99.5 99.5 | 98.8 98.8 98.9 99.5 99.5 | 0.0 0.0 0.0 0.7 0.7 | 0.0 0.0 0.0 69.6 99.7 |
-x-EUC-TW 26788 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 0.0 0.0 0.0 0.1 0.1 | 81.0 81.0 81.0 81.1 81.1 |
-x-MacRoman 1994 | 63.0 63.0 81.0 73.6 73.7 | 63.4 63.4 81.5 74.0 74.2 | 63.5 63.5 81.7 74.1 74.2 | 63.5 63.5 81.7 74.1 74.2 | 0.0 0.0 0.0 10.6 10.8 | 0.0 0.0 0.0 10.6 10.8 |
-x-mac-cyrillic 1773 | 70.8 70.8 70.8 70.8 70.8 | 70.8 70.8 70.8 70.8 70.8 | 70.8 70.8 70.8 70.8 70.8 | 70.8 70.8 70.8 70.8 70.8 | 0.0 0.0 0.0 0.0 0.0 | 78.1 78.1 78.1 78.1 78.1 |
-x-windows-949 36719 | 99.1 99.1 99.1 99.1 99.1 | 98.9 98.9 98.9 99.1 99.1 | 98.9 98.9 98.9 99.1 99.1 | 98.9 98.9 98.9 99.1 99.1 | 0.0 99.4 99.5 99.4 99.4 | 0.0 99.5 99.5 99.6 99.6 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1085250 | 79.0 82.4 83.8 85.0 85.0 | 92.3 95.7 96.7 97.9 97.9 | 92.3 95.7 96.7 97.9 97.9 | 91.7 95.1 95.9 97.9 97.9 | 45.5 65.1 70.4 63.2 63.4 | 34.7 49.5 49.5 50.3 52.1 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 32.3 | 28.5 | 27.9 | 27.7 | 65.8 | 11.1 |
-
-=== Probe length: 200B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 99.9 99.9 51.5 51.6 | 0.0 84.5 84.5 51.3 51.3 |
-EUC-JP 37043 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.7 99.7 99.7 99.8 99.8 | 99.4 99.4 99.5 99.6 99.6 | 98.8 98.8 98.8 98.9 98.9 |
-EUC-KR 36883 | 0.0 99.5 99.5 99.5 99.5 | 0.0 99.5 99.5 99.5 99.5 | 0.0 99.5 99.5 99.5 99.5 | 0.0 99.5 99.5 99.5 99.5 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 |
-GB18030 36862 | 99.5 99.5 99.5 99.7 99.7 | 99.5 99.5 99.5 99.7 99.7 | 99.5 99.5 99.5 99.7 99.7 | 99.5 99.5 99.5 99.7 99.7 | 98.5 98.5 99.3 98.7 98.7 | 99.3 99.3 99.3 99.4 99.4 |
-IBM500 31455 | 100.0 100.0 100.0 100.0 100.0 | 86.1 86.1 86.1 86.1 86.1 | 86.1 86.1 86.1 86.1 86.1 | 86.1 86.1 86.1 86.1 86.1 | 88.5 88.5 100.0 88.5 88.5 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 30539 | 94.3 94.3 96.0 98.4 98.4 | 94.5 94.5 96.1 98.6 98.6 | 94.5 94.5 96.1 98.6 98.6 | 94.5 94.5 96.1 98.6 98.6 | 0.0 0.0 0.0 3.3 3.4 | 0.0 0.0 0.0 3.3 3.4 |
-IBM852 35403 | 97.9 97.9 98.5 99.3 99.3 | 98.0 98.0 98.5 99.3 99.3 | 98.0 98.0 98.5 99.3 99.3 | 98.0 98.0 98.5 99.3 99.3 | 0.0 0.0 0.0 1.1 1.1 | 0.0 0.0 0.0 1.1 1.1 |
-IBM855 36702 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 99.9 99.9 99.9 99.9 99.9 |
-IBM866 36985 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 97.4 97.4 99.7 97.5 97.5 | 99.9 99.9 99.9 99.9 99.9 |
-ISO-8859-16 32899 | 94.8 94.8 99.7 99.5 99.5 | 94.7 94.7 95.0 99.4 99.5 | 94.8 94.8 95.0 99.5 99.6 | 91.6 91.6 91.7 99.5 99.6 | 0.0 0.0 0.0 47.3 47.6 | 0.0 0.0 0.0 40.5 40.7 |
-ISO-8859-3 35648 | 99.1 99.1 99.4 99.6 99.6 | 99.1 99.1 99.4 99.6 99.6 | 99.1 99.1 99.4 99.6 99.6 | 99.1 99.1 99.4 99.6 99.6 | 0.0 0.0 0.0 0.6 0.6 | 0.0 0.0 0.0 0.6 0.6 |
-KOI8-R 36850 | 99.7 100.0 100.0 100.0 100.0 | 99.7 100.0 100.0 100.0 100.0 | 99.7 100.0 100.0 100.0 100.0 | 99.7 100.0 100.0 100.0 100.0 | 98.4 98.4 99.8 98.4 98.4 | 99.9 99.9 99.9 99.9 99.9 |
-KOI8-U 36846 | 99.8 100.0 100.0 99.9 99.9 | 99.8 100.0 100.0 99.9 99.9 | 99.8 100.0 100.0 99.9 99.9 | 99.8 100.0 100.0 99.9 99.9 | 0.0 95.9 99.6 0.2 0.2 | 0.0 99.8 99.8 0.2 0.2 |
-Shift_JIS 36917 | 99.8 99.8 99.8 99.9 99.9 | 98.4 98.4 98.4 98.5 98.5 | 98.4 98.4 98.4 98.5 98.5 | 98.4 98.4 98.4 98.5 98.5 | 99.7 99.7 99.7 99.8 99.8 | 99.6 99.6 99.6 99.8 99.8 |
-UTF-16-BE 36799 | 0.0 0.0 0.0 0.0 0.0 | 96.0 96.0 96.0 96.0 96.0 | 96.0 96.0 96.0 96.0 96.0 | 96.0 96.0 96.0 96.0 96.0 | 69.3 69.3 95.0 69.3 69.3 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 0.0 0.0 0.0 0.0 0.0 | 96.0 96.0 96.0 96.0 96.0 | 96.0 96.0 96.0 96.0 96.0 | 96.0 96.0 96.0 96.0 96.0 | 69.6 69.6 95.2 69.6 69.6 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 97.5 97.5 99.3 99.1 99.1 | 98.4 98.4 98.4 100.0 100.0 | 98.4 98.4 98.4 100.0 100.0 | 98.4 98.4 98.4 100.0 100.0 | 98.0 98.0 98.7 99.6 99.6 | 97.9 97.9 97.9 99.4 99.4 |
-windows-1250 34499 | 91.5 91.5 92.6 97.0 97.4 | 91.5 91.5 92.6 97.0 97.4 | 91.5 91.5 92.6 97.0 97.5 | 90.1 90.1 91.1 97.0 97.5 | 60.6 78.3 99.5 81.3 81.3 | 0.0 0.0 0.0 5.5 7.6 |
-windows-1251 36852 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 97.1 97.1 99.0 97.1 97.1 | 89.7 89.7 89.7 89.7 89.7 |
-windows-1252 25874 | 79.2 79.2 82.1 93.0 93.2 | 86.9 86.9 89.9 93.1 93.2 | 87.0 87.0 89.9 93.2 93.3 | 92.7 92.7 95.9 93.2 93.3 | 21.8 90.2 99.5 94.2 94.2 | 0.0 99.1 99.1 75.7 98.9 |
-windows-1253 36845 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 16.1 99.5 99.8 95.8 95.9 | 0.4 99.4 99.4 80.0 95.3 |
-windows-1254 36705 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 37.1 98.5 100.0 98.6 98.6 | 0.0 0.0 0.0 0.1 0.2 |
-windows-1255 31252 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 41.9 77.8 90.3 71.0 77.8 | 99.7 99.7 99.7 99.7 99.8 |
-windows-1256 41912 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 48.9 86.1 98.7 49.0 49.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1257 30789 | 96.6 96.6 98.3 97.4 97.4 | 96.7 96.7 98.3 97.4 97.4 | 96.7 96.7 98.3 97.4 97.4 | 94.2 94.2 94.9 97.4 97.4 | 0.0 0.0 0.0 10.7 10.7 | 0.0 0.0 0.0 8.6 11.9 |
-windows-1258 36885 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 0.0 0.0 0.0 0.1 0.1 | 0.0 0.0 0.0 0.1 0.1 |
-windows-874 31440 | 99.7 99.7 99.7 99.9 99.9 | 99.7 99.7 99.7 99.9 99.9 | 99.7 99.7 99.7 99.9 99.9 | 99.7 99.7 99.7 99.9 99.9 | 0.0 0.0 0.0 0.2 0.2 | 0.0 0.0 0.0 54.5 99.9 |
-x-EUC-TW 26788 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 81.1 81.1 81.1 81.1 81.1 |
-x-MacRoman 1994 | 86.3 86.3 92.2 88.8 88.8 | 86.6 86.6 92.5 89.0 89.0 | 86.6 86.6 92.5 89.1 89.1 | 86.6 86.6 92.5 89.1 89.1 | 0.0 0.0 0.0 2.5 2.5 | 0.0 0.0 0.0 2.5 2.5 |
-x-mac-cyrillic 1773 | 88.2 88.2 88.2 88.2 88.2 | 88.2 88.2 88.2 88.2 88.2 | 88.2 88.2 88.2 88.2 88.2 | 88.2 88.2 88.2 88.2 88.2 | 0.0 0.0 0.0 0.0 0.0 | 85.7 85.7 85.7 85.7 85.7 |
-x-windows-949 36719 | 99.6 99.6 99.6 99.6 99.6 | 99.5 99.5 99.5 99.6 99.6 | 99.5 99.5 99.5 99.6 99.6 | 99.5 99.5 99.5 99.6 99.6 | 0.0 99.9 99.9 99.6 99.6 | 0.0 99.8 99.8 99.6 99.6 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1085250 | 81.5 84.9 85.4 85.8 85.9 | 94.6 98.0 98.2 98.7 98.8 | 94.6 98.0 98.2 98.7 98.8 | 94.5 97.9 98.1 98.7 98.8 | 48.2 67.2 71.4 63.0 63.2 | 35.0 49.9 49.9 47.8 50.4 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 41.3 | 37.8 | 36.7 | 36.5 | 102.0 | 15.0 |
-
-=== Probe length: full ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 99.9 100.0 33.8 33.8 | 0.0 84.6 84.6 33.7 33.7 |
-EUC-JP 37043 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.9 99.9 99.9 99.9 99.9 | 99.6 99.6 99.6 99.6 99.6 |
-EUC-KR 36883 | 0.0 99.7 99.7 99.7 99.7 | 0.0 99.7 99.7 99.7 99.7 | 0.0 99.7 99.7 99.7 99.7 | 0.0 99.7 99.7 99.7 99.7 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 |
-GB18030 36862 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.5 99.5 99.8 99.5 99.5 | 99.7 99.7 99.7 99.7 99.7 |
-IBM500 31455 | 100.0 100.0 100.0 100.0 100.0 | 91.4 91.4 91.4 91.4 91.4 | 91.4 91.4 91.4 91.4 91.4 | 91.4 91.4 91.4 91.4 91.4 | 72.2 72.2 100.0 72.2 72.2 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 30539 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM852 35403 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM855 36702 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 |
-IBM866 36985 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 98.9 98.9 99.9 98.9 98.9 | 100.0 100.0 100.0 100.0 100.0 |
-ISO-8859-16 32899 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.5 99.5 99.5 99.8 99.8 | 0.0 0.0 0.0 14.0 14.3 | 0.0 0.0 0.0 11.9 12.1 |
-ISO-8859-3 35648 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-KOI8-R 36850 | 99.9 100.0 100.0 100.0 100.0 | 99.9 100.0 100.0 100.0 100.0 | 99.9 100.0 100.0 100.0 100.0 | 99.9 100.0 100.0 100.0 100.0 | 99.3 99.3 99.9 99.3 99.3 | 99.9 99.9 99.9 99.9 99.9 |
-KOI8-U 36846 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 0.0 98.2 99.8 0.1 0.1 | 0.0 99.9 99.9 0.1 0.1 |
-Shift_JIS 36917 | 100.0 100.0 100.0 100.0 100.0 | 99.5 99.5 99.5 99.5 99.5 | 99.5 99.5 99.5 99.5 99.5 | 99.5 99.5 99.5 99.5 99.5 | 100.0 100.0 100.0 100.0 100.0 | 99.9 99.9 99.9 99.9 99.9 |
-UTF-16-BE 36799 | 0.0 0.0 0.0 0.0 0.0 | 95.6 95.6 95.6 95.6 95.6 | 95.6 95.6 95.6 95.6 95.6 | 95.6 95.6 95.6 95.6 95.6 | 68.6 68.6 95.7 68.6 68.6 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 0.0 0.0 0.0 0.0 0.0 | 95.6 95.6 95.6 95.6 95.6 | 95.6 95.6 95.6 95.6 95.6 | 95.6 95.6 95.6 95.6 95.6 | 68.8 68.8 96.5 68.8 68.8 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 99.9 99.9 99.9 99.9 99.9 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.6 99.6 99.6 99.6 99.6 |
-windows-1250 34499 | 99.1 99.1 99.2 99.1 99.1 | 99.1 99.1 99.2 99.1 99.1 | 99.1 99.1 99.2 99.1 99.1 | 99.1 99.1 99.2 99.1 99.1 | 80.3 85.5 99.9 84.5 84.5 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1251 36852 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 98.9 98.9 99.6 98.9 98.9 | 93.2 93.2 93.2 93.2 93.2 |
-windows-1252 25874 | 99.5 99.5 99.5 99.5 99.5 | 99.5 99.5 99.5 99.5 99.5 | 99.5 99.5 99.5 99.5 99.5 | 99.5 99.5 99.5 99.5 99.5 | 46.7 94.4 99.8 94.4 94.4 | 0.0 98.9 98.9 50.7 98.2 |
-windows-1253 36845 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 33.7 99.8 99.9 93.8 93.9 | 1.0 99.7 99.7 61.1 90.4 |
-windows-1254 36705 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 60.3 99.3 100.0 99.3 99.3 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1255 31252 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 84.2 96.3 99.1 84.3 96.3 | 99.8 99.9 99.9 99.8 99.9 |
-windows-1256 41912 | 99.8 99.8 99.8 99.8 99.8 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 52.0 91.6 99.4 52.0 52.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1257 30789 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1258 36885 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-874 31440 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 11.2 99.9 |
-x-EUC-TW 26788 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 81.1 81.1 81.1 81.1 81.1 |
-x-MacRoman 1994 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-x-mac-cyrillic 1773 | 99.4 99.4 99.4 99.4 99.4 | 99.4 99.4 99.4 99.4 99.4 | 99.4 99.4 99.4 99.4 99.4 | 99.4 99.4 99.4 99.4 99.4 | 0.0 0.0 0.0 0.0 0.0 | 95.4 95.4 95.4 95.4 95.4 |
-x-windows-949 36719 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 0.0 100.0 100.0 99.3 99.3 | 0.0 99.9 99.9 99.3 99.3 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1085250 | 82.9 86.3 86.3 86.3 86.3 | 95.9 99.3 99.4 99.3 99.3 | 95.9 99.3 99.4 99.3 99.3 | 95.9 99.3 99.3 99.3 99.3 | 51.8 68.1 71.9 61.2 61.6 | 35.3 50.2 50.2 43.6 48.3 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 64.5 | 60.6 | 58.4 | 58.4 | 227.6 | 29.8 |
-
-=== Accuracy by probe length (All detector) ===
- Length Strict% Soft% Top3% Decode% Alpha%
- ----------------------------------------------------------
- 20B 71.9 75.8 78.1 90.6 90.6
- 50B 85.3 88.8 90.2 95.9 95.9
- 100B 91.7 95.1 95.9 97.9 97.9
- 200B 94.5 97.9 98.1 98.7 98.8
- full 95.9 99.3 99.3 99.3 99.3
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-eval-v6-no-utf32.txt b/docs/modules/ROOT/pages/advanced/charset-detection-eval-v6-no-utf32.txt
deleted file mode 100644
index bed925d67a..0000000000
--- a/docs/modules/ROOT/pages/advanced/charset-detection-eval-v6-no-utf32.txt
+++ /dev/null
@@ -1,387 +0,0 @@
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-=== Probe length: 8B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-SLF4J(W): No SLF4J providers were found.
-SLF4J(W): Defaulting to no-operation (NOP) logger implementation
-SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
-Big5-HKSCS 30334 | 74.0 74.0 94.1 78.3 78.3 | 73.6 73.6 89.9 81.5 81.6 | 77.3 77.3 89.9 85.0 85.1 | 77.3 77.3 89.9 85.0 85.1 | 0.0 51.5 88.5 56.2 56.5 | 0.0 0.0 0.0 7.7 8.8 |
-EUC-JP 37043 | 62.1 62.1 77.0 79.4 80.4 | 61.9 61.9 75.2 82.0 83.0 | 61.7 61.7 74.4 81.2 82.6 | 61.7 61.7 74.4 81.2 82.6 | 0.0 0.0 37.7 16.3 19.1 | 0.0 0.0 0.0 17.6 20.0 |
-EUC-KR 36883 | 66.5 66.5 86.3 73.1 73.3 | 68.0 68.0 85.6 75.5 75.7 | 67.8 67.8 85.2 75.2 75.4 | 67.8 67.8 85.2 75.2 75.4 | 0.0 0.0 63.1 6.9 7.3 | 0.0 0.0 0.0 7.3 7.8 |
-GB18030 36862 | 72.1 72.1 81.9 84.2 84.9 | 71.7 71.7 80.6 85.6 86.3 | 71.8 71.8 80.5 85.5 86.3 | 71.8 71.8 80.5 85.5 86.3 | 0.4 0.4 7.3 13.6 14.6 | 1.0 1.0 1.0 14.3 15.6 |
-IBM1047 34790 | 6.9 54.4 69.1 54.3 54.4 | 7.2 55.8 70.0 55.8 55.8 | 7.3 56.0 70.3 56.0 56.0 | 7.3 56.0 70.3 56.0 56.0 | 0.0 78.7 83.5 78.5 78.6 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-ltr 36874 | 74.5 94.4 97.1 74.5 74.5 | 74.6 94.7 96.7 74.6 74.6 | 74.7 94.7 96.7 74.7 74.7 | 74.7 94.7 96.7 74.7 74.7 | 0.0 55.6 62.6 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-rtl 37018 | 73.2 92.5 97.3 73.2 73.2 | 73.8 93.3 97.1 73.8 73.8 | 73.9 93.4 97.1 73.9 73.9 | 73.9 93.4 97.1 73.9 73.9 | 0.0 65.2 70.5 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-ltr 34927 | 76.2 93.0 96.6 76.2 76.2 | 9.5 14.2 17.6 9.5 9.5 | 9.6 14.3 17.7 9.6 9.6 | 9.6 14.3 17.7 9.6 9.6 | 0.0 51.4 56.6 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-rtl 32984 | 82.5 93.4 97.4 82.5 82.5 | 12.3 13.3 16.8 12.3 12.3 | 12.3 13.4 16.8 12.3 12.3 | 12.3 13.4 16.8 12.3 12.3 | 0.0 56.3 59.5 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM500 35083 | 47.6 54.4 69.3 54.4 54.4 | 48.7 55.7 70.2 55.7 55.7 | 48.9 56.0 70.5 56.0 56.0 | 48.9 56.0 70.5 56.0 56.0 | 78.7 78.7 83.4 78.7 78.7 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 34505 | 29.0 29.0 58.2 91.8 92.0 | 17.0 17.0 22.0 94.3 94.5 | 17.1 17.1 22.2 94.5 94.5 | 17.1 17.1 22.2 94.5 94.5 | 0.0 0.0 0.0 71.8 72.4 | 0.0 0.0 0.0 75.5 76.3 |
-IBM852 35418 | 46.3 46.3 72.0 91.2 91.3 | 24.4 24.4 30.6 93.5 93.6 | 24.6 24.6 30.8 93.7 93.7 | 24.6 24.6 30.8 93.7 93.7 | 0.0 0.0 0.0 62.0 62.1 | 0.0 0.0 0.0 67.0 67.2 |
-IBM855 36702 | 65.8 65.8 90.4 70.4 70.4 | 68.3 68.3 90.6 73.2 73.2 | 73.8 73.8 91.5 78.7 78.7 | 73.8 73.8 91.5 78.7 78.7 | 0.0 0.0 0.0 4.6 4.7 | 79.7 79.7 79.7 84.6 84.6 |
-IBM866 36985 | 76.3 76.3 90.9 81.1 81.1 | 80.7 80.7 91.4 85.8 85.8 | 82.5 82.5 92.2 87.6 87.6 | 82.5 82.5 92.2 87.6 87.6 | 30.5 30.5 49.2 35.4 35.4 | 88.9 88.9 88.9 94.1 94.1 |
-ISO-2022-CN 40954 | 0.0 0.0 0.0 13.6 17.0 | 81.4 81.4 81.4 99.1 100.0 | 81.4 81.4 81.4 99.1 100.0 | 81.4 81.4 81.4 99.1 100.0 | 36.5 36.5 79.2 50.2 54.3 | 0.0 0.0 0.0 14.2 18.2 |
-ISO-2022-JP 37151 | 0.0 0.0 0.0 16.1 19.3 | 80.5 80.5 80.5 98.2 99.9 | 80.5 80.5 80.5 98.2 99.9 | 80.5 80.5 80.5 98.2 99.9 | 69.9 69.9 73.6 86.4 89.7 | 78.6 78.6 78.6 96.4 100.0 |
-ISO-2022-KR 36860 | 0.0 0.0 0.0 7.0 9.3 | 89.7 89.7 89.7 99.3 100.0 | 89.7 89.7 89.7 99.3 100.0 | 89.7 89.7 89.7 99.3 100.0 | 88.0 88.0 89.6 94.9 97.7 | 89.7 89.7 89.7 97.1 100.0 |
-ISO-8859-16 32901 | 49.7 49.7 66.2 93.8 94.0 | 20.9 20.9 24.6 95.5 95.6 | 21.0 21.0 24.7 95.5 95.6 | 21.0 21.0 24.7 95.5 95.6 | 0.0 0.0 0.0 82.8 83.2 | 0.0 0.0 0.0 91.4 92.1 |
-ISO-8859-3 35648 | 46.5 46.5 75.6 91.8 91.9 | 18.4 18.4 22.3 95.1 95.1 | 18.6 18.6 22.5 95.4 95.4 | 18.6 18.6 22.5 95.4 95.4 | 0.0 0.0 0.0 72.2 72.2 | 0.0 0.0 0.0 76.8 76.8 |
-KOI8-R 36850 | 67.4 81.8 93.3 86.6 86.7 | 67.4 81.8 92.8 87.0 87.0 | 68.2 82.8 93.0 88.0 88.1 | 68.2 82.8 93.0 88.0 88.1 | 40.9 40.9 47.2 45.9 45.9 | 90.7 90.7 90.7 95.9 96.0 |
-KOI8-U 36846 | 57.6 83.4 94.9 85.6 85.6 | 57.6 83.4 94.6 85.8 85.9 | 59.0 85.3 95.0 87.7 87.7 | 59.0 85.3 95.0 87.7 87.7 | 0.0 33.7 40.1 24.7 24.7 | 0.0 89.2 89.2 61.1 61.1 |
-Shift_JIS 36917 | 60.2 60.2 75.3 76.7 76.9 | 63.8 63.8 76.3 81.6 81.8 | 63.6 63.6 75.5 81.1 81.2 | 63.6 63.6 75.5 81.1 81.2 | 0.0 0.0 15.3 16.4 17.3 | 0.0 0.0 0.0 17.4 18.6 |
-US-ASCII 36759 | 0.0 5.7 18.2 97.2 97.2 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 9.2 75.4 95.8 95.8 | 0.0 0.0 0.0 100.0 100.0 |
-UTF-16-BE 36799 | 70.9 71.5 84.2 70.9 70.9 | 92.4 92.5 96.8 92.4 92.4 | 95.2 95.3 97.4 95.2 95.2 | 95.2 95.3 97.4 95.2 95.2 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 69.2 69.5 83.9 69.2 69.2 | 95.1 95.1 97.4 95.1 95.1 | 95.6 95.6 97.8 95.6 95.6 | 95.6 95.6 97.8 95.6 95.6 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 90.1 90.1 99.8 90.1 90.1 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 96.9 96.9 100.0 96.9 96.9 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 64.8 64.8 76.7 86.5 86.6 | 62.2 62.2 69.2 87.3 87.4 | 62.3 62.3 69.3 87.3 87.4 | 62.3 62.3 69.3 87.3 87.4 | 60.2 60.2 93.1 79.6 79.6 | 69.0 69.0 69.0 94.0 94.0 |
-windows-1250 34548 | 20.6 20.6 35.0 88.3 88.6 | 20.0 20.0 26.3 90.3 90.3 | 20.0 20.0 26.4 90.4 90.4 | 20.0 20.0 26.4 90.4 90.4 | 4.5 40.3 63.0 80.0 80.3 | 0.0 0.0 0.0 78.3 81.6 |
-windows-1251 36852 | 77.5 77.5 87.7 82.0 82.1 | 77.6 77.6 87.7 82.4 82.4 | 77.8 77.8 88.0 82.6 82.6 | 77.8 77.8 88.0 82.6 82.6 | 35.1 35.4 41.1 39.8 39.9 | 64.8 65.6 65.6 69.6 69.8 |
-windows-1252 25975 | 8.6 8.6 16.1 81.2 81.7 | 77.1 77.1 84.3 83.3 83.4 | 77.1 77.1 84.3 83.3 83.4 | 77.1 77.1 84.3 83.3 83.4 | 1.8 42.3 76.9 82.9 83.2 | 0.0 98.9 98.9 94.9 98.9 |
-windows-1253 36845 | 75.3 75.3 88.6 82.4 82.5 | 75.1 75.1 87.6 82.8 82.9 | 75.4 75.4 87.9 83.1 83.1 | 75.4 75.4 87.9 83.1 83.1 | 0.7 38.0 52.9 45.1 45.1 | 0.0 75.6 75.6 81.5 83.0 |
-windows-1254 36705 | 40.6 40.6 60.8 86.9 87.1 | 33.8 33.8 42.1 89.2 89.2 | 34.0 34.0 42.3 89.4 89.4 | 34.0 34.0 42.3 89.4 89.4 | 1.9 32.9 45.8 74.1 74.3 | 0.0 0.0 0.0 68.0 71.9 |
-windows-1255 36774 | 75.8 75.8 93.5 79.1 79.1 | 75.8 75.8 93.4 79.3 79.3 | 75.9 75.9 93.5 79.3 79.3 | 75.9 75.9 93.5 79.3 79.3 | 0.7 15.5 20.1 18.6 18.8 | 80.9 82.0 82.0 85.3 85.5 |
-windows-1256 41912 | 89.5 89.5 95.6 92.5 92.6 | 89.5 89.5 95.3 92.8 92.8 | 90.1 90.1 95.5 93.3 93.3 | 90.1 90.1 95.5 93.3 93.3 | 27.6 39.5 47.4 30.7 30.7 | 0.0 0.0 0.0 3.2 3.3 |
-windows-1257 35316 | 48.0 48.0 73.0 88.4 88.7 | 25.5 25.5 34.5 89.8 89.8 | 25.6 25.6 34.6 89.8 89.9 | 25.6 25.6 34.6 89.8 89.9 | 0.0 0.0 0.0 64.2 64.6 | 0.0 0.0 0.0 68.0 70.7 |
-windows-1258 36885 | 69.4 69.4 81.7 88.0 89.5 | 63.5 63.5 72.0 88.9 89.9 | 64.3 64.3 72.3 89.7 90.2 | 64.3 64.3 72.3 89.7 90.2 | 0.0 0.0 0.0 25.0 26.0 | 0.0 0.0 0.0 24.6 26.0 |
-windows-874 36809 | 58.2 58.2 77.1 74.6 74.7 | 60.2 60.2 74.8 78.6 78.7 | 62.0 62.0 75.9 80.5 80.5 | 62.0 62.0 75.9 80.5 80.5 | 0.0 0.0 0.0 17.5 17.6 | 0.0 0.0 0.0 86.8 89.0 |
-x-EUC-TW 26788 | 69.2 69.2 89.1 75.9 76.1 | 69.4 69.4 86.3 77.3 77.5 | 69.0 69.0 85.6 76.6 77.0 | 69.0 69.0 85.6 76.6 77.0 | 0.0 0.0 0.0 7.6 8.3 | 0.0 0.0 0.0 7.6 8.3 |
-x-MacRoman 36756 | 29.0 29.0 62.8 90.0 90.3 | 17.2 17.2 22.1 92.7 92.8 | 17.5 17.5 22.3 93.0 93.1 | 17.5 17.5 22.3 93.0 93.1 | 0.0 0.0 0.0 72.1 72.4 | 0.0 0.0 0.0 75.5 75.9 |
-x-mac-cyrillic 36631 | 77.5 77.5 88.4 77.5 77.5 | 77.7 77.7 88.5 77.7 77.7 | 77.9 77.9 88.7 77.9 77.9 | 77.9 77.9 88.7 77.9 77.9 | 0.0 0.0 0.0 0.0 0.0 | 42.1 42.1 42.1 42.1 42.1 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1469647 | 50.8 54.9 68.1 71.5 71.9 | 58.6 62.1 70.0 82.7 82.9 | 59.1 62.6 70.2 83.2 83.4 | 59.1 62.6 70.2 83.2 83.4 | 16.7 30.2 42.1 44.0 44.5 | 17.2 23.1 23.1 46.1 47.1 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 10.4 | 6.4 | 6.3 | 6.2 | 5.1 | 2.2 |
-
---- Confusion (All/ML+rules, 8B, top errors per charset) ---
- Big5-HKSCS 22.7% wrong → windows-1252:7.7%, GB18030:7.5%, EUC-JP:1.0%, EUC-KR:0.9%, KOI8-U:0.6%
- EUC-JP 38.3% wrong → windows-1252:17.6%, GB18030:6.3%, EUC-KR:4.5%, x-EUC-TW:1.8%, Big5-HKSCS:1.3%
- EUC-KR 32.2% wrong → GB18030:8.4%, windows-1252:8.2%, EUC-JP:5.4%, windows-1251:3.0%, Big5-HKSCS:1.4%
- GB18030 28.2% wrong → windows-1252:13.6%, EUC-JP:2.2%, windows-1251:1.9%, EUC-KR:1.7%, x-EUC-TW:1.2%
- IBM1047 92.7% wrong → IBM500:48.8%, IBM424-rtl:12.1%, IBM424-ltr:6.2%, IBM420-ltr:5.2%, GB18030:3.2%
- IBM420-ltr 25.3% wrong → IBM420-rtl:20.1%, IBM500:1.1%, IBM424-rtl:0.9%, IBM424-ltr:0.8%, windows-1252:0.7%
- IBM420-rtl 26.1% wrong → IBM420-ltr:19.5%, IBM424-rtl:3.6%, windows-1252:0.5%, IBM500:0.4%, IBM424-ltr:0.4%
- IBM424-ltr 90.4% wrong → windows-1252:79.3%, IBM424-rtl:4.7%, IBM500:2.3%, IBM420-ltr:0.8%, GB18030:0.4%
- IBM424-rtl 87.7% wrong → windows-1252:80.9%, IBM420-rtl:1.3%, IBM500:1.1%, IBM424-ltr:1.0%, IBM420-ltr:0.7%
- IBM500 51.1% wrong → IBM424-rtl:12.5%, IBM1047:7.1%, IBM424-ltr:6.1%, IBM420-ltr:5.1%, GB18030:3.4%
- IBM850 82.9% wrong → windows-1252:75.6%, IBM852:2.4%, x-MacRoman:0.8%, IBM424-rtl:0.5%, ISO-8859-16:0.5%
- IBM852 75.4% wrong → windows-1252:67.1%, IBM850:2.8%, x-MacRoman:0.8%, IBM424-rtl:0.5%, windows-1256:0.5%
- IBM855 26.2% wrong → GB18030:8.0%, windows-1252:5.0%, IBM850:3.0%, windows-1256:2.0%, IBM866:1.2%
- IBM866 17.5% wrong → windows-1252:5.2%, IBM850:3.1%, Big5-HKSCS:1.6%, x-mac-cyrillic:1.3%, GB18030:1.1%
- ISO-2022-CN 18.6% wrong → windows-1252:15.2%, ISO-2022-JP:3.4%, UTF-16BE:0.0%, UTF-16-BE:0.0%, IBM424-rtl:0.0%
- ISO-2022-JP 19.5% wrong → windows-1252:19.5%
- ISO-2022-KR 10.3% wrong → windows-1252:8.1%, ISO-2022-JP:2.3%
- ISO-8859-16 79.0% wrong → windows-1252:73.1%, windows-1250:1.6%, windows-1257:1.4%, IBM852:0.6%, IBM850:0.3%
- ISO-8859-3 81.4% wrong → windows-1252:76.4%, x-MacRoman:1.2%, IBM850:0.7%, ISO-8859-16:0.4%, UTF-16-LE:0.3%
- KOI8-R 31.8% wrong → KOI8-U:14.6%, windows-1252:5.3%, windows-1253:3.7%, windows-1251:2.3%, windows-1256:2.1%
- KOI8-U 41.0% wrong → KOI8-R:26.3%, windows-1252:3.4%, GB18030:2.9%, windows-1253:2.4%, windows-1256:2.0%
- Shift_JIS 36.4% wrong → windows-1252:17.4%, x-mac-cyrillic:2.9%, GB18030:2.5%, x-MacRoman:2.3%, IBM424-rtl:2.2%
- US-ASCII 100.0% wrong → windows-1252:100.0%
- UTF-16-BE 4.8% wrong → windows-1252:0.8%, windows-1256:0.4%, windows-1253:0.4%, IBM424-rtl:0.3%, IBM852:0.2%
- UTF-16-LE 4.4% wrong → windows-1252:0.8%, IBM420-ltr:0.5%, IBM850:0.3%, IBM852:0.2%, GB18030:0.2%
- UTF-8 37.7% wrong → windows-1252:25.6%, GB18030:4.5%, IBM850:1.7%, IBM866:1.1%, windows-1256:0.8%
- windows-1250 80.0% wrong → windows-1252:65.1%, ISO-8859-16:3.5%, x-MacRoman:2.1%, windows-1257:2.0%, IBM852:1.5%
- windows-1251 22.2% wrong → x-mac-cyrillic:5.3%, windows-1252:5.1%, windows-1253:3.0%, windows-1256:2.2%, windows-1250:1.1%
- windows-1252 22.9% wrong → windows-1257:5.4%, windows-1250:2.8%, IBM850:2.3%, windows-1254:2.0%, x-MacRoman:1.6%
- windows-1253 24.6% wrong → windows-1252:9.7%, windows-1256:4.7%, KOI8-R:2.0%, windows-1251:1.4%, KOI8-U:1.1%
- windows-1254 66.0% wrong → windows-1252:53.7%, IBM852:2.5%, windows-1257:2.4%, IBM850:1.2%, x-MacRoman:1.1%
- windows-1255 24.1% wrong → windows-1252:5.7%, windows-1253:4.0%, windows-1251:3.0%, windows-1257:2.9%, windows-1250:2.3%
- windows-1256 9.9% wrong → windows-1252:3.7%, windows-1253:1.6%, windows-1251:1.3%, KOI8-R:0.6%, windows-1250:0.6%
- windows-1257 74.4% wrong → windows-1252:62.7%, IBM850:2.0%, ISO-8859-16:1.8%, IBM852:1.6%, windows-1254:1.4%
- windows-1258 35.7% wrong → windows-1252:24.7%, windows-1257:2.0%, IBM850:1.9%, windows-874:1.0%, x-MacRoman:0.8%
- windows-874 38.0% wrong → windows-1252:18.4%, GB18030:5.5%, EUC-JP:3.5%, windows-1251:2.1%, KOI8-U:1.1%
- x-EUC-TW 31.0% wrong → windows-1252:7.6%, GB18030:7.1%, windows-1256:2.8%, KOI8-R:2.0%, windows-1251:1.8%
- x-MacRoman 82.5% wrong → windows-1252:75.5%, IBM850:2.4%, IBM852:0.9%, windows-1256:0.5%, windows-1250:0.3%
- x-mac-cyrillic 22.1% wrong → windows-1252:5.5%, windows-1251:5.3%, windows-1253:2.1%, windows-1256:1.8%, windows-1250:1.4%
-
-=== Probe length: 32B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 94.6 94.6 97.3 97.3 97.3 | 94.6 94.6 96.9 97.5 97.6 | 94.9 94.9 96.9 97.8 97.9 | 94.9 94.9 96.9 97.8 97.9 | 0.0 78.1 86.5 65.3 65.4 | 0.0 70.2 70.2 67.6 67.8 |
-EUC-JP 37043 | 89.2 89.2 94.9 92.9 93.2 | 89.2 89.2 94.7 93.2 93.5 | 89.2 89.2 94.6 93.2 93.5 | 89.2 89.2 94.6 93.2 93.5 | 61.5 61.5 66.9 65.2 65.7 | 79.8 79.8 79.8 83.5 84.0 |
-EUC-KR 36883 | 92.9 92.9 97.1 94.2 94.3 | 93.0 93.0 97.1 94.4 94.4 | 93.0 93.0 97.1 94.4 94.4 | 93.0 93.0 97.1 94.4 94.4 | 78.9 78.9 83.2 80.2 80.3 | 92.7 92.7 92.7 94.0 94.2 |
-GB18030 36862 | 93.6 93.6 95.5 96.6 96.8 | 93.5 93.5 95.0 96.9 97.1 | 93.6 93.6 95.1 96.9 97.1 | 93.6 93.6 95.1 96.9 97.1 | 59.8 59.8 66.2 63.0 63.5 | 71.7 71.7 71.7 75.0 75.5 |
-IBM1047 34790 | 12.6 90.0 95.1 89.0 89.9 | 12.6 90.2 95.2 89.1 90.1 | 12.7 90.2 95.2 89.1 90.1 | 12.7 90.2 95.2 89.1 90.1 | 0.0 82.4 98.6 80.9 82.1 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-ltr 36874 | 96.1 99.0 99.5 96.1 96.1 | 96.2 99.0 99.5 96.2 96.2 | 96.2 99.0 99.5 96.2 96.2 | 96.2 99.0 99.5 96.2 96.2 | 0.0 91.9 94.6 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-rtl 37018 | 96.0 99.1 99.8 96.0 96.0 | 96.0 99.1 99.8 96.0 96.0 | 96.0 99.1 99.8 96.0 96.0 | 96.0 99.1 99.8 96.0 96.0 | 0.0 94.2 96.1 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-ltr 34927 | 92.9 97.1 98.9 92.9 92.9 | 34.2 37.7 39.5 34.2 34.2 | 34.2 37.7 39.5 34.2 34.2 | 34.2 37.7 39.5 34.2 34.2 | 0.0 85.4 90.3 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-rtl 32984 | 96.4 99.6 99.9 96.4 96.4 | 29.0 29.7 30.0 29.0 29.0 | 29.0 29.7 30.0 29.0 29.0 | 29.0 29.7 30.0 29.0 29.0 | 0.0 89.6 92.3 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM500 35083 | 78.1 90.7 95.4 90.6 90.7 | 78.2 90.8 95.5 90.7 90.8 | 78.2 90.8 95.5 90.7 90.8 | 78.2 90.8 95.5 90.7 90.8 | 82.8 82.8 98.7 82.8 82.8 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 34505 | 59.3 59.3 84.9 93.7 93.9 | 49.7 49.7 56.5 94.0 94.2 | 49.7 49.7 56.5 94.0 94.2 | 49.7 49.7 56.5 94.0 94.2 | 0.0 0.0 0.0 41.8 42.7 | 0.0 0.0 0.0 42.1 43.1 |
-IBM852 35418 | 80.6 80.6 93.2 94.8 94.9 | 66.7 66.7 71.8 95.1 95.1 | 66.8 66.8 71.8 95.2 95.2 | 66.8 66.8 71.8 95.2 95.2 | 0.0 0.0 0.0 26.3 26.6 | 0.0 0.0 0.0 26.9 27.2 |
-IBM855 36702 | 95.0 95.0 98.9 95.6 95.7 | 95.1 95.1 98.9 95.8 95.9 | 97.9 97.9 99.0 98.6 98.6 | 97.9 97.9 99.0 98.6 98.6 | 0.0 0.0 0.0 0.7 0.7 | 97.1 97.1 97.1 97.8 97.8 |
-IBM866 36985 | 97.9 97.9 99.0 98.7 98.7 | 98.1 98.1 99.0 98.9 98.9 | 98.3 98.3 99.0 99.1 99.1 | 98.3 98.3 99.0 99.1 99.1 | 63.9 63.9 86.6 64.7 64.7 | 97.7 97.7 97.7 98.5 98.5 |
-ISO-2022-CN 40954 | 0.0 0.0 0.0 4.0 4.5 | 95.4 95.4 95.4 99.8 100.0 | 95.4 95.4 95.4 99.8 100.0 | 95.4 95.4 95.4 99.8 100.0 | 92.5 92.5 94.3 96.5 97.1 | 0.0 0.0 0.0 4.0 4.6 |
-ISO-2022-JP 37151 | 0.0 0.0 0.0 3.8 4.2 | 95.8 95.8 95.8 99.7 100.0 | 95.8 95.8 95.8 99.7 100.0 | 95.8 95.8 95.8 99.7 100.0 | 86.5 86.5 89.2 90.3 90.7 | 95.7 95.7 95.7 99.6 100.0 |
-ISO-2022-KR 36860 | 0.0 0.0 0.0 1.5 1.7 | 98.3 98.3 98.3 99.9 100.0 | 98.3 98.3 98.3 99.9 100.0 | 98.3 98.3 98.3 99.9 100.0 | 96.4 96.4 96.6 97.9 98.1 | 98.3 98.3 98.3 99.8 100.0 |
-ISO-8859-16 32901 | 69.1 69.1 86.1 94.2 94.3 | 51.6 51.6 61.9 93.9 93.9 | 51.6 51.6 62.0 93.9 93.9 | 51.6 51.6 62.0 93.9 93.9 | 0.0 0.0 0.0 80.2 80.6 | 0.0 0.0 0.0 75.5 76.0 |
-ISO-8859-3 35648 | 85.1 85.1 95.8 96.6 96.7 | 61.4 61.4 64.3 96.9 96.9 | 61.5 61.5 64.3 97.0 97.0 | 61.5 61.5 64.3 97.0 97.0 | 0.0 0.0 0.0 35.5 35.5 | 0.0 0.0 0.0 36.3 36.3 |
-KOI8-R 36850 | 95.4 98.2 99.0 98.9 98.9 | 95.3 98.2 99.0 98.9 98.9 | 95.4 98.3 99.0 99.0 99.0 | 95.4 98.3 99.0 99.0 99.0 | 76.4 76.4 86.2 77.1 77.1 | 98.3 98.3 98.3 99.0 99.0 |
-KOI8-U 36846 | 87.8 98.3 99.4 96.3 96.4 | 87.8 98.2 99.3 96.4 96.4 | 88.3 98.8 99.3 96.9 96.9 | 88.3 98.8 99.3 96.9 96.9 | 0.0 68.4 81.0 11.4 11.4 | 0.0 98.4 98.4 17.8 17.8 |
-Shift_JIS 36917 | 89.6 89.6 94.7 93.1 93.2 | 91.7 91.7 94.9 95.3 95.4 | 91.7 91.7 94.9 95.4 95.5 | 91.7 91.7 94.9 95.4 95.5 | 65.1 65.1 65.8 68.7 69.0 | 85.3 85.3 85.3 89.0 89.4 |
-US-ASCII 36759 | 0.0 1.2 4.3 99.6 99.6 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.4 59.7 99.5 99.5 | 0.0 0.0 0.0 100.0 100.0 |
-UTF-16-BE 36799 | 84.8 85.2 95.3 84.8 84.8 | 98.3 98.3 99.3 98.3 98.3 | 99.0 99.1 99.7 99.0 99.0 | 99.0 99.1 99.7 99.0 99.0 | 71.0 71.0 92.8 71.0 71.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 93.5 93.5 98.5 93.5 93.5 | 99.1 99.1 99.7 99.1 99.1 | 99.5 99.5 99.9 99.5 99.5 | 99.5 99.5 99.9 99.5 99.5 | 72.0 72.0 93.1 72.0 72.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 81.8 81.8 87.4 93.6 93.6 | 87.7 87.7 87.7 100.0 100.0 | 87.7 87.7 87.7 100.0 100.0 | 87.7 87.7 87.7 100.0 100.0 | 86.0 86.0 92.8 98.0 98.0 | 86.9 86.9 86.9 99.2 99.2 |
-windows-1250 34548 | 58.7 58.7 78.6 91.1 91.4 | 56.7 56.7 67.8 91.0 91.2 | 56.7 56.7 67.8 91.0 91.2 | 56.7 56.7 67.8 91.0 91.2 | 17.2 59.4 91.3 82.4 82.4 | 0.0 0.0 0.0 45.1 48.9 |
-windows-1251 36852 | 93.6 93.6 98.2 94.5 94.5 | 93.6 93.6 98.2 94.5 94.5 | 93.7 93.7 98.3 94.6 94.6 | 93.7 93.7 98.3 94.6 94.6 | 69.5 69.5 80.6 70.4 70.4 | 79.8 79.9 79.9 80.7 80.8 |
-windows-1252 25975 | 46.3 46.3 65.3 88.3 88.7 | 82.5 82.5 91.6 88.4 88.6 | 82.5 82.5 91.7 88.4 88.6 | 82.5 82.5 91.7 88.4 88.6 | 5.6 74.2 95.9 89.5 89.5 | 0.0 98.6 98.6 91.3 98.6 |
-windows-1253 36845 | 95.4 95.4 97.6 96.9 96.9 | 95.4 95.4 97.6 96.9 96.9 | 95.4 95.4 97.6 96.9 96.9 | 95.4 95.4 97.6 96.9 96.9 | 3.4 84.6 92.4 85.0 85.1 | 0.1 93.7 93.7 90.0 94.0 |
-windows-1254 36705 | 86.6 86.6 94.8 94.5 94.6 | 83.0 83.0 88.4 94.4 94.5 | 83.1 83.1 88.4 94.5 94.5 | 83.1 83.1 88.4 94.5 94.5 | 8.6 69.7 90.6 81.2 81.2 | 0.0 0.0 0.0 22.6 26.0 |
-windows-1255 36774 | 97.9 97.9 99.1 98.5 98.6 | 97.8 97.8 99.1 98.5 98.6 | 97.9 97.9 99.1 98.6 98.6 | 97.9 97.9 99.1 98.6 98.6 | 4.8 34.5 47.9 34.6 35.3 | 96.7 98.3 98.3 98.8 99.0 |
-windows-1256 41912 | 98.9 98.9 99.1 99.6 99.6 | 98.9 98.9 99.1 99.6 99.6 | 98.9 98.9 99.1 99.6 99.6 | 98.9 98.9 99.1 99.6 99.6 | 39.2 66.3 86.4 40.0 40.0 | 0.0 0.0 0.0 0.7 0.8 |
-windows-1257 35316 | 82.0 82.0 94.1 93.3 93.4 | 69.0 69.0 76.2 93.0 93.1 | 69.0 69.0 76.2 93.1 93.1 | 69.0 69.0 76.2 93.1 93.1 | 0.0 0.0 0.0 36.3 36.3 | 0.0 0.0 0.0 36.0 38.5 |
-windows-1258 36885 | 96.3 96.3 98.0 98.1 98.3 | 96.0 96.0 97.4 98.1 98.3 | 96.2 96.2 97.4 98.2 98.4 | 96.2 96.2 97.4 98.2 98.4 | 0.0 0.0 0.0 2.1 2.1 | 0.0 0.0 0.0 2.0 2.1 |
-windows-874 36809 | 89.5 89.5 94.3 94.8 94.8 | 89.8 89.8 93.6 95.3 95.3 | 91.4 91.4 93.7 96.9 97.0 | 91.4 91.4 93.7 96.9 97.0 | 0.0 0.0 0.0 5.5 5.5 | 0.0 0.0 0.0 93.2 97.7 |
-x-EUC-TW 26788 | 92.7 92.7 96.9 95.5 95.6 | 92.6 92.6 96.6 95.6 95.6 | 92.6 92.6 96.6 95.6 95.6 | 92.6 92.6 96.6 95.6 95.6 | 0.0 0.0 0.0 2.9 3.0 | 68.3 68.3 68.3 71.2 71.3 |
-x-MacRoman 36756 | 64.9 64.9 89.4 92.8 93.1 | 52.2 52.2 57.8 93.1 93.4 | 52.2 52.2 57.8 93.1 93.4 | 52.2 52.2 57.8 93.1 93.4 | 0.0 0.0 0.0 40.6 41.0 | 0.0 0.0 0.0 41.0 41.4 |
-x-mac-cyrillic 36631 | 92.7 92.7 98.3 92.7 92.7 | 92.8 92.8 98.3 92.8 92.8 | 92.8 92.8 98.4 92.8 92.8 | 92.8 92.8 98.4 92.8 92.8 | 0.0 0.0 0.0 0.0 0.0 | 58.5 58.5 58.5 58.5 58.5 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1469647 | 72.0 74.8 80.4 82.8 82.9 | 80.7 83.4 86.3 93.2 93.3 | 80.8 83.6 86.3 93.4 93.5 | 80.8 83.6 86.3 93.4 93.5 | 33.9 55.1 63.8 54.8 55.0 | 29.8 37.8 37.8 49.2 49.9 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 12.1 | 8.3 | 8.2 | 8.0 | 14.1 | 3.2 |
-
---- Confusion (All/ML+rules, 32B, top errors per charset) ---
- Big5-HKSCS 5.1% wrong → windows-1252:2.9%, GB18030:1.6%, IBM850:0.1%, ISO-8859-16:0.1%, IBM852:0.1%
- EUC-JP 10.8% wrong → GB18030:4.2%, windows-1252:3.8%, EUC-KR:0.6%, x-EUC-TW:0.3%, IBM850:0.2%
- EUC-KR 7.0% wrong → GB18030:2.5%, windows-1252:1.9%, EUC-JP:1.1%, windows-1251:0.3%, Big5-HKSCS:0.1%
- GB18030 6.4% wrong → windows-1252:3.4%, windows-1251:0.3%, EUC-JP:0.3%, UTF-8:0.3%, IBM850:0.3%
- IBM1047 87.3% wrong → IBM500:77.6%, IBM424-rtl:3.6%, IBM420-ltr:2.0%, IBM424-ltr:1.3%, IBM852:0.5%
- IBM420-ltr 3.8% wrong → IBM420-rtl:2.8%, IBM500:0.6%, IBM424-ltr:0.1%, IBM424-rtl:0.1%, IBM1047:0.1%
- IBM420-rtl 4.0% wrong → IBM420-ltr:3.1%, IBM424-rtl:0.8%, IBM500:0.0%, GB18030:0.0%, x-mac-cyrillic:0.0%
- IBM424-ltr 65.8% wrong → windows-1252:58.6%, IBM424-rtl:3.5%, IBM500:1.9%, UTF-8:0.8%, IBM420-ltr:0.3%
- IBM424-rtl 71.0% wrong → windows-1252:69.7%, IBM424-ltr:0.6%, UTF-8:0.2%, IBM500:0.2%, IBM420-rtl:0.1%
- IBM500 21.8% wrong → IBM1047:12.6%, IBM424-rtl:3.4%, IBM420-ltr:1.8%, IBM424-ltr:1.2%, IBM852:0.4%
- IBM850 50.3% wrong → windows-1252:42.4%, IBM852:3.0%, x-MacRoman:1.8%, ISO-8859-16:0.9%, windows-1257:0.8%
- IBM852 33.2% wrong → windows-1252:27.1%, IBM850:2.4%, ISO-8859-16:0.8%, windows-1250:0.8%, x-MacRoman:0.7%
- IBM855 2.1% wrong → windows-1252:0.7%, GB18030:0.4%, IBM850:0.3%, windows-1256:0.1%, IBM852:0.1%
- IBM866 1.7% wrong → windows-1252:0.7%, x-mac-cyrillic:0.3%, IBM850:0.2%, x-MacRoman:0.1%, IBM852:0.1%
- ISO-2022-CN 4.6% wrong → windows-1252:4.2%, ISO-2022-JP:0.5%, UTF-16BE:0.0%
- ISO-2022-JP 4.2% wrong → windows-1252:4.2%
- ISO-2022-KR 1.7% wrong → windows-1252:1.6%, ISO-2022-JP:0.1%, UTF-16BE:0.0%
- ISO-8859-16 48.4% wrong → windows-1252:34.8%, windows-1250:8.6%, windows-1257:1.8%, IBM852:0.8%, UTF-8:0.8%
- ISO-8859-3 38.5% wrong → windows-1252:35.4%, x-MacRoman:0.7%, ISO-8859-16:0.4%, IBM852:0.4%, windows-1257:0.4%
- KOI8-R 4.6% wrong → KOI8-U:2.9%, windows-1252:0.8%, windows-1253:0.2%, windows-1251:0.1%, windows-1256:0.1%
- KOI8-U 11.7% wrong → KOI8-R:10.6%, windows-1252:0.5%, windows-1253:0.1%, windows-1251:0.1%, windows-1256:0.1%
- Shift_JIS 8.3% wrong → windows-1252:3.7%, x-MacRoman:0.7%, GB18030:0.7%, IBM852:0.6%, x-mac-cyrillic:0.6%
- US-ASCII 100.0% wrong → windows-1252:100.0%
- UTF-16-BE 1.0% wrong → windows-1256:0.1%, IBM850:0.1%, x-MacRoman:0.1%, IBM420-rtl:0.1%, windows-1250:0.1%
- UTF-16-LE 0.5% wrong → windows-1253:0.1%, windows-1251:0.0%, UTF-16BE:0.0%, IBM420-rtl:0.0%, KOI8-U:0.0%
- UTF-8 12.3% wrong → windows-1252:12.3%
- windows-1250 43.3% wrong → windows-1252:26.9%, ISO-8859-16:7.3%, IBM852:3.2%, x-MacRoman:1.7%, windows-1257:1.6%
- windows-1251 6.3% wrong → x-mac-cyrillic:4.0%, windows-1252:1.1%, windows-1257:0.2%, windows-1253:0.1%, windows-1255:0.1%
- windows-1252 17.5% wrong → x-MacRoman:3.5%, windows-1250:3.1%, windows-1257:2.8%, IBM850:2.3%, windows-1254:1.4%
- windows-1253 4.6% wrong → windows-1252:2.7%, KOI8-R:0.4%, windows-1256:0.3%, windows-1257:0.2%, windows-1254:0.2%
- windows-1254 16.9% wrong → windows-1252:11.6%, IBM852:1.1%, windows-1250:0.8%, windows-1257:0.8%, ISO-8859-16:0.6%
- windows-1255 2.1% wrong → windows-1252:1.2%, windows-1250:0.2%, windows-1257:0.2%, windows-1251:0.1%, windows-1253:0.1%
- windows-1256 1.1% wrong → windows-1252:0.8%, windows-1253:0.1%, UTF-8:0.1%, x-MacRoman:0.0%, windows-1250:0.0%
- windows-1257 31.0% wrong → windows-1252:22.8%, windows-1250:1.5%, IBM852:1.4%, windows-1254:1.2%, ISO-8859-16:1.1%
- windows-1258 3.8% wrong → windows-1252:2.0%, x-MacRoman:0.3%, ISO-8859-16:0.2%, windows-1257:0.2%, IBM850:0.2%
- windows-874 8.6% wrong → windows-1252:5.4%, GB18030:0.8%, windows-1251:0.3%, windows-1257:0.2%, EUC-JP:0.2%
- x-EUC-TW 7.4% wrong → GB18030:3.2%, windows-1252:2.9%, UTF-8:0.2%, windows-1256:0.2%, Big5-HKSCS:0.1%
- x-MacRoman 47.8% wrong → windows-1252:41.2%, IBM850:2.7%, IBM852:1.2%, ISO-8859-16:0.7%, windows-1250:0.5%
- x-mac-cyrillic 7.2% wrong → windows-1251:4.8%, windows-1252:1.0%, windows-1255:0.2%, windows-1253:0.2%, x-MacRoman:0.2%
-
-=== Probe length: 128B ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 99.8 99.8 99.9 99.9 99.9 | 99.8 99.8 99.9 99.9 99.9 | 99.8 99.8 99.9 99.9 99.9 | 99.8 99.8 99.9 99.9 99.9 | 0.0 99.6 99.8 58.9 58.9 | 0.0 84.3 84.3 58.6 58.6 |
-EUC-JP 37043 | 98.5 98.5 98.7 98.8 98.8 | 98.5 98.5 98.7 98.8 98.8 | 98.5 98.5 98.7 98.8 98.8 | 98.5 98.5 98.7 98.8 98.8 | 98.6 98.6 98.8 99.0 99.0 | 97.7 97.7 97.7 98.1 98.1 |
-EUC-KR 36883 | 99.6 99.6 99.6 99.6 99.6 | 99.6 99.6 99.6 99.6 99.6 | 99.6 99.6 99.6 99.6 99.6 | 99.6 99.6 99.6 99.6 99.6 | 99.7 99.7 99.7 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 |
-GB18030 36862 | 99.1 99.1 99.2 99.5 99.5 | 99.1 99.1 99.2 99.5 99.6 | 99.1 99.1 99.2 99.5 99.6 | 99.1 99.1 99.2 99.5 99.6 | 97.0 97.0 98.3 97.4 97.5 | 98.4 98.4 98.4 98.8 98.9 |
-IBM1047 34790 | 10.4 98.5 98.8 92.9 98.1 | 10.4 98.6 98.8 93.0 98.2 | 10.4 98.6 98.8 93.0 98.2 | 10.4 98.6 98.8 93.0 98.2 | 0.0 88.9 100.0 82.6 88.1 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-ltr 36874 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 98.4 99.6 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-rtl 37018 | 99.7 99.8 99.8 99.7 99.7 | 99.7 99.8 99.8 99.7 99.7 | 99.7 99.8 99.8 99.7 99.7 | 99.7 99.8 99.8 99.7 99.7 | 0.0 98.6 99.7 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-ltr 34927 | 99.5 99.7 99.7 99.5 99.5 | 70.9 71.1 71.1 70.9 70.9 | 70.9 71.1 71.1 70.9 70.9 | 70.9 71.1 71.1 70.9 70.9 | 0.0 96.3 99.1 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-rtl 32984 | 99.9 100.0 100.0 99.9 99.9 | 56.8 56.9 56.9 56.8 56.8 | 56.8 56.9 56.9 56.8 56.8 | 56.8 56.9 56.9 56.8 56.8 | 0.0 96.3 98.9 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM500 35083 | 89.5 98.7 98.9 98.4 98.7 | 89.5 98.7 99.0 98.4 98.7 | 89.5 98.7 99.0 98.4 98.7 | 89.5 98.7 99.0 98.4 98.7 | 89.0 89.0 100.0 89.0 89.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 34505 | 88.5 88.5 93.9 94.7 94.7 | 87.3 87.3 90.7 94.7 94.7 | 87.3 87.3 90.7 94.7 94.7 | 87.3 87.3 90.7 94.7 94.7 | 0.0 0.0 0.0 7.0 7.3 | 0.0 0.0 0.0 7.0 7.3 |
-IBM852 35418 | 97.7 97.7 98.5 99.1 99.1 | 96.2 96.2 96.7 99.1 99.1 | 96.2 96.2 96.7 99.1 99.1 | 96.2 96.2 96.7 99.1 99.1 | 0.0 0.0 0.0 2.7 2.8 | 0.0 0.0 0.0 2.7 2.8 |
-IBM855 36702 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 99.8 99.8 99.8 99.9 99.9 |
-IBM866 36985 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 94.5 94.5 99.5 94.6 94.6 | 99.8 99.8 99.8 99.9 99.9 |
-ISO-2022-CN 40954 | 0.0 0.0 0.0 0.8 0.8 | 99.2 99.2 99.2 100.0 100.0 | 99.2 99.2 99.2 100.0 100.0 | 99.2 99.2 99.2 100.0 100.0 | 98.3 98.3 98.6 99.1 99.2 | 0.0 0.0 0.0 0.8 0.8 |
-ISO-2022-JP 37151 | 0.0 0.0 0.0 0.5 0.5 | 99.5 99.5 99.5 100.0 100.0 | 99.5 99.5 99.5 100.0 100.0 | 99.5 99.5 99.5 100.0 100.0 | 99.1 99.1 99.3 99.6 99.7 | 99.5 99.5 99.5 100.0 100.0 |
-ISO-2022-KR 36860 | 0.0 0.0 0.0 0.1 0.1 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.2 99.2 99.3 99.3 99.3 | 99.9 99.9 99.9 100.0 100.0 |
-ISO-8859-16 32901 | 83.2 83.2 89.0 96.3 96.3 | 77.5 77.5 82.4 96.3 96.3 | 77.5 77.5 82.4 96.3 96.3 | 77.5 77.5 82.4 96.3 96.3 | 0.0 0.0 0.0 57.8 58.1 | 0.0 0.0 0.0 49.9 50.1 |
-ISO-8859-3 35648 | 99.3 99.3 99.7 99.6 99.6 | 97.5 97.5 97.8 99.6 99.6 | 97.5 97.5 97.8 99.6 99.6 | 97.5 97.5 97.8 99.6 99.6 | 0.0 0.0 0.0 2.4 2.4 | 0.0 0.0 0.0 2.4 2.4 |
-KOI8-R 36850 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 97.3 97.3 99.6 97.3 97.3 | 99.8 99.8 99.8 99.8 99.8 |
-KOI8-U 36846 | 98.9 99.9 99.9 99.2 99.2 | 98.9 99.9 99.9 99.2 99.2 | 98.9 99.9 99.9 99.2 99.2 | 98.9 99.9 99.9 99.2 99.2 | 0.0 94.2 99.1 0.4 0.4 | 0.0 99.7 99.7 0.5 0.5 |
-Shift_JIS 36917 | 99.2 99.2 99.3 99.5 99.5 | 99.2 99.2 99.3 99.6 99.6 | 99.2 99.2 99.3 99.6 99.6 | 99.2 99.2 99.3 99.6 99.6 | 99.0 99.0 99.0 99.3 99.3 | 99.1 99.1 99.1 99.5 99.5 |
-US-ASCII 36759 | 0.0 0.0 0.1 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 65.8 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 |
-UTF-16-BE 36799 | 88.6 89.1 91.7 88.6 88.6 | 98.4 98.4 98.8 98.4 98.4 | 98.8 98.8 98.8 98.8 98.8 | 98.8 98.8 98.8 98.8 98.8 | 69.0 69.0 94.4 69.0 69.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 94.2 94.2 96.1 94.2 94.2 | 99.0 99.0 99.2 99.0 99.0 | 99.1 99.1 99.3 99.1 99.1 | 99.1 99.1 99.3 99.1 99.1 | 69.6 69.6 94.6 69.6 69.6 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 92.7 92.7 93.6 95.4 95.4 | 97.3 97.3 97.3 100.0 100.0 | 97.3 97.3 97.3 100.0 100.0 | 97.3 97.3 97.3 100.0 100.0 | 96.5 96.5 98.0 99.3 99.3 | 96.6 96.6 96.6 99.3 99.3 |
-windows-1250 34548 | 88.6 88.6 90.7 95.5 95.7 | 88.5 88.5 90.4 95.5 95.6 | 88.5 88.5 90.4 95.5 95.6 | 88.5 88.5 90.4 95.5 95.6 | 49.6 74.4 98.9 80.3 80.3 | 0.0 0.0 0.0 10.6 13.5 |
-windows-1251 36852 | 98.2 98.2 98.5 98.3 98.3 | 98.2 98.2 98.5 98.3 98.3 | 98.2 98.2 98.5 98.3 98.3 | 98.2 98.2 98.5 98.3 98.3 | 94.7 94.7 97.8 94.7 94.7 | 88.2 88.2 88.2 88.2 88.2 |
-windows-1252 25975 | 89.4 89.4 93.7 95.2 95.3 | 94.0 94.0 96.6 95.1 95.2 | 94.0 94.0 96.6 95.1 95.2 | 94.0 94.0 96.6 95.1 95.2 | 16.0 88.0 99.3 93.8 93.8 | 0.0 99.1 99.1 81.5 99.0 |
-windows-1253 36845 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 11.9 99.0 99.7 96.1 96.2 | 0.3 98.9 98.9 84.5 95.9 |
-windows-1254 36705 | 99.6 99.6 99.8 99.7 99.7 | 99.5 99.5 99.7 99.7 99.7 | 99.5 99.5 99.7 99.7 99.7 | 99.5 99.5 99.7 99.7 99.7 | 29.0 96.3 99.8 96.5 96.5 | 0.0 0.0 0.0 0.6 0.8 |
-windows-1255 36774 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 100.0 | 99.9 99.9 99.9 99.9 100.0 | 99.9 99.9 99.9 99.9 100.0 | 14.3 60.4 77.9 58.1 60.5 | 99.6 99.7 99.7 99.8 99.8 |
-windows-1256 41912 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 47.8 82.4 97.5 47.9 47.9 | 0.0 0.0 0.0 0.1 0.1 |
-windows-1257 35316 | 98.3 98.3 99.3 99.2 99.2 | 97.4 97.4 98.4 99.2 99.2 | 97.4 97.4 98.4 99.2 99.2 | 97.4 97.4 98.4 99.2 99.2 | 0.0 0.0 0.0 15.0 15.0 | 0.0 0.0 0.0 14.9 16.5 |
-windows-1258 36885 | 99.8 99.8 99.9 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 99.8 99.8 99.8 99.9 99.9 | 0.0 0.0 0.0 0.1 0.1 | 0.0 0.0 0.0 0.1 0.1 |
-windows-874 36809 | 99.4 99.4 99.5 99.8 99.8 | 99.4 99.4 99.5 99.8 99.8 | 99.4 99.4 99.5 99.9 99.9 | 99.4 99.4 99.5 99.9 99.9 | 0.0 0.0 0.0 0.4 0.4 | 0.0 0.0 0.0 88.3 99.9 |
-x-EUC-TW 26788 | 99.4 99.4 99.5 99.5 99.5 | 99.4 99.4 99.5 99.5 99.5 | 99.4 99.4 99.5 99.5 99.5 | 99.4 99.4 99.5 99.5 99.5 | 0.0 0.0 0.0 0.1 0.1 | 81.1 81.1 81.1 81.2 81.2 |
-x-MacRoman 36756 | 92.2 92.2 96.8 96.1 96.2 | 90.1 90.1 92.7 96.1 96.2 | 90.1 90.1 92.7 96.1 96.2 | 90.1 90.1 92.7 96.1 96.2 | 0.0 0.0 0.0 6.1 6.2 | 0.0 0.0 0.0 6.1 6.2 |
-x-mac-cyrillic 36631 | 99.7 99.7 99.8 99.7 99.7 | 99.7 99.7 99.8 99.7 99.7 | 99.7 99.7 99.8 99.7 99.7 | 99.7 99.7 99.8 99.7 99.7 | 0.0 0.0 0.0 0.0 0.0 | 76.4 76.4 76.4 76.4 76.4 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1469647 | 80.0 82.4 83.1 85.7 85.9 | 91.4 93.7 94.2 97.3 97.5 | 91.4 93.8 94.2 97.4 97.5 | 91.4 93.8 94.2 97.4 97.5 | 42.0 65.8 71.6 56.9 57.2 | 32.9 41.4 41.4 44.7 45.8 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 11.4 | 8.8 | 8.4 | 8.3 | 40.3 | 5.8 |
-
---- Confusion (All/ML+rules, 128B, top errors per charset) ---
- Big5-HKSCS 0.2% wrong → GB18030:0.1%, windows-1252:0.0%, IBM850:0.0%, ISO-8859-16:0.0%, windows-874:0.0%
- EUC-JP 1.5% wrong → GB18030:0.9%, windows-1252:0.3%, IBM850:0.1%, x-MacRoman:0.1%, IBM852:0.0%
- EUC-KR 0.4% wrong → windows-1252:0.2%, GB18030:0.1%, x-MacRoman:0.0%, EUC-JP:0.0%, IBM850:0.0%
- GB18030 0.9% wrong → windows-1252:0.5%, IBM850:0.1%, x-MacRoman:0.1%, ISO-8859-3:0.0%, ISO-8859-16:0.0%
- IBM1047 89.6% wrong → IBM500:88.2%, IBM420-ltr:0.7%, IBM424-rtl:0.3%, UTF-16-BE:0.2%, IBM424-ltr:0.1%
- IBM420-ltr 0.1% wrong → IBM500:0.1%, IBM420-rtl:0.0%, UTF-16-BE:0.0%, IBM424-ltr:0.0%, IBM1047:0.0%
- IBM420-rtl 0.3% wrong → IBM424-rtl:0.2%, IBM420-ltr:0.1%
- IBM424-ltr 29.1% wrong → windows-1252:28.4%, IBM500:0.2%, UTF-8:0.2%, IBM424-rtl:0.2%, IBM420-ltr:0.0%
- IBM424-rtl 43.2% wrong → windows-1252:43.0%, UTF-8:0.1%, IBM424-ltr:0.0%, IBM500:0.0%, windows-1253:0.0%
- IBM500 10.5% wrong → IBM1047:9.2%, IBM420-ltr:0.6%, IBM424-rtl:0.2%, UTF-16-BE:0.2%, IBM424-ltr:0.1%
- IBM850 12.7% wrong → windows-1252:9.2%, x-MacRoman:1.9%, IBM852:0.7%, windows-1257:0.4%, ISO-8859-16:0.4%
- IBM852 3.8% wrong → windows-1252:2.8%, ISO-8859-16:0.3%, IBM850:0.3%, windows-1250:0.2%, x-MacRoman:0.1%
- IBM855 0.1% wrong → windows-1252:0.1%, IBM852:0.0%, x-MacRoman:0.0%, windows-1250:0.0%, UTF-8:0.0%
- IBM866 0.1% wrong → windows-1252:0.1%, x-mac-cyrillic:0.0%, IBM852:0.0%, ISO-8859-3:0.0%, IBM850:0.0%
- ISO-2022-CN 0.8% wrong → windows-1252:0.8%, ISO-2022-JP:0.0%
- ISO-2022-JP 0.5% wrong → windows-1252:0.5%
- ISO-2022-KR 0.1% wrong → windows-1252:0.1%, ISO-2022-JP:0.0%
- ISO-8859-16 22.5% wrong → windows-1250:12.9%, windows-1252:8.5%, IBM852:0.4%, windows-1257:0.2%, x-MacRoman:0.2%
- ISO-8859-3 2.5% wrong → windows-1252:2.2%, x-MacRoman:0.1%, IBM852:0.0%, UTF-8:0.0%, ISO-8859-16:0.0%
- KOI8-R 0.1% wrong → windows-1252:0.1%, KOI8-U:0.0%, x-MacRoman:0.0%, IBM850:0.0%, windows-1253:0.0%
- KOI8-U 1.1% wrong → KOI8-R:1.0%, windows-1252:0.0%, x-MacRoman:0.0%, windows-1251:0.0%, ISO-8859-16:0.0%
- Shift_JIS 0.8% wrong → windows-1252:0.3%, x-MacRoman:0.1%, IBM850:0.1%, IBM852:0.1%, x-mac-cyrillic:0.0%
- US-ASCII 100.0% wrong → windows-1252:100.0%, ISO-2022-JP:0.0%
- UTF-16-BE 1.2% wrong → windows-1252:1.0%, UTF-8:0.0%, UTF-16LE:0.0%, ISO-8859-16:0.0%, IBM500:0.0%
- UTF-16-LE 0.9% wrong → windows-1252:0.7%, KOI8-U:0.1%, IBM866:0.0%, IBM852:0.0%, UTF-16BE:0.0%
- UTF-8 2.7% wrong → windows-1252:2.7%
- windows-1250 11.5% wrong → ISO-8859-16:5.6%, windows-1252:3.2%, IBM852:2.0%, windows-1257:0.3%, x-MacRoman:0.2%
- windows-1251 1.8% wrong → x-mac-cyrillic:1.6%, windows-1252:0.1%, windows-1250:0.0%, windows-1257:0.0%, IBM852:0.0%
- windows-1252 6.0% wrong → x-MacRoman:2.6%, IBM850:1.0%, windows-1257:0.5%, windows-1250:0.4%, ISO-8859-16:0.3%
- windows-1253 0.2% wrong → windows-1252:0.2%, KOI8-R:0.0%, ISO-8859-3:0.0%, windows-1257:0.0%, x-MacRoman:0.0%
- windows-1254 0.5% wrong → windows-1252:0.4%, IBM852:0.0%, windows-1250:0.0%, IBM850:0.0%, ISO-8859-16:0.0%
- windows-1255 0.1% wrong → windows-1252:0.1%, x-MacRoman:0.0%, IBM852:0.0%, IBM850:0.0%, windows-1254:0.0%
- windows-1256 0.1% wrong → windows-1252:0.1%, x-MacRoman:0.0%, UTF-8:0.0%, windows-1254:0.0%, ISO-8859-3:0.0%
- windows-1257 2.6% wrong → windows-1252:2.0%, x-MacRoman:0.1%, windows-1250:0.1%, IBM852:0.1%, windows-1254:0.1%
- windows-1258 0.2% wrong → windows-1252:0.2%, x-MacRoman:0.0%, IBM852:0.0%, IBM850:0.0%, ISO-8859-3:0.0%
- windows-874 0.6% wrong → windows-1252:0.4%, x-MacRoman:0.0%, windows-1254:0.0%, IBM850:0.0%, GB18030:0.0%
- x-EUC-TW 0.6% wrong → GB18030:0.5%, windows-1252:0.1%, windows-1256:0.0%, Big5-HKSCS:0.0%, x-MacRoman:0.0%
- x-MacRoman 9.9% wrong → windows-1252:7.8%, IBM850:1.3%, ISO-8859-16:0.3%, IBM852:0.2%, windows-1257:0.1%
- x-mac-cyrillic 0.3% wrong → windows-1251:0.2%, windows-1252:0.0%, IBM852:0.0%, x-MacRoman:0.0%, IBM866:0.0%
-
-=== Probe length: full ===
- N | --- ML ablation --------------------------------------------------- | --- Baselines --------------------------------- |
-Charset | Stat R% S% T3% D% A% | +ISO R% S% T3% D% A% | +CJK R% S% T3% D% A% | All R% S% T3% D% A% | ICU4J R% S% T3% D% A% | juniv R% S% T3% D% A% |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 30334 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 99.9 100.0 33.8 33.8 | 0.0 84.6 84.6 33.7 33.7 |
-EUC-JP 37043 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.8 99.8 99.8 99.8 99.8 | 99.9 99.9 99.9 99.9 99.9 | 99.6 99.6 99.6 99.6 99.6 |
-EUC-KR 36883 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 |
-GB18030 36862 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.5 99.5 99.8 99.5 99.5 | 99.7 99.7 99.7 99.7 99.7 |
-IBM1047 34790 | 15.2 99.6 99.7 82.2 98.1 | 15.2 99.7 99.7 82.2 98.2 | 15.2 99.7 99.7 82.2 98.2 | 15.2 99.7 99.7 82.2 98.2 | 0.0 75.7 100.0 59.7 73.6 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-ltr 36874 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 99.5 99.9 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM420-rtl 37018 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 99.9 100.0 100.0 99.9 99.9 | 0.0 99.3 99.9 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-ltr 34927 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 94.3 99.7 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM424-rtl 32984 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 92.0 99.6 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM500 35083 | 89.0 99.7 99.8 98.3 99.7 | 89.1 99.8 99.8 98.4 99.8 | 89.1 99.8 99.8 98.4 99.8 | 89.1 99.8 99.8 98.4 99.8 | 76.1 76.1 100.0 76.1 76.1 | 0.0 0.0 0.0 0.0 0.0 |
-IBM850 34505 | 99.7 99.7 99.9 99.8 99.8 | 99.7 99.7 99.9 99.8 99.8 | 99.7 99.7 99.9 99.8 99.8 | 99.7 99.7 99.9 99.8 99.8 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM852 35418 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-IBM855 36702 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 |
-IBM866 36985 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 98.9 98.9 99.9 98.9 98.9 | 100.0 100.0 100.0 100.0 100.0 |
-ISO-2022-CN 40954 | 0.0 0.0 0.0 0.3 0.3 | 99.7 99.7 99.7 100.0 100.0 | 99.7 99.7 99.7 100.0 100.0 | 99.7 99.7 99.7 100.0 100.0 | 99.4 99.4 99.4 99.6 99.6 | 0.0 0.0 0.0 0.3 0.3 |
-ISO-2022-JP 37151 | 0.0 0.0 0.0 0.1 0.1 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.9 99.9 99.9 100.0 100.0 | 99.8 99.8 99.8 99.9 99.9 | 99.9 99.9 99.9 100.0 100.0 |
-ISO-2022-KR 36860 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.7 99.7 99.7 99.7 99.7 | 100.0 100.0 100.0 100.0 100.0 |
-ISO-8859-16 32901 | 95.4 95.4 96.5 97.5 97.5 | 95.4 95.4 96.5 97.5 97.5 | 95.4 95.4 96.5 97.5 97.5 | 95.4 95.4 96.5 97.5 97.5 | 0.0 0.0 0.0 14.1 14.4 | 0.0 0.0 0.0 11.9 12.2 |
-ISO-8859-3 35648 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-KOI8-R 36850 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.3 99.3 99.9 99.3 99.3 | 99.9 99.9 99.9 99.9 99.9 |
-KOI8-U 36846 | 99.7 100.0 100.0 99.8 99.8 | 99.7 100.0 100.0 99.8 99.8 | 99.7 100.0 100.0 99.8 99.8 | 99.7 100.0 100.0 99.8 99.8 | 0.0 98.2 99.8 0.1 0.1 | 0.0 99.9 99.9 0.1 0.1 |
-Shift_JIS 36917 | 99.9 99.9 100.0 99.9 99.9 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.9 99.9 99.9 99.9 99.9 |
-US-ASCII 36759 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 | 0.0 0.0 69.4 100.0 100.0 | 0.0 0.0 0.0 100.0 100.0 |
-UTF-16-BE 36799 | 93.7 94.4 95.1 93.7 93.7 | 98.7 98.7 98.9 98.7 98.7 | 98.8 98.9 98.9 98.8 98.8 | 98.8 98.9 98.9 98.8 98.8 | 68.6 68.6 95.7 68.6 68.6 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-16-LE 36736 | 97.5 97.5 98.1 97.5 97.5 | 99.3 99.3 99.5 99.3 99.3 | 99.4 99.4 99.5 99.4 99.4 | 99.4 99.4 99.5 99.4 99.4 | 68.8 68.8 96.5 68.8 68.8 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-BE 36757 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-32-LE 37011 | 0.0 0.0 0.0 0.0 0.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 |
-UTF-8 36254 | 98.8 98.8 99.0 98.8 98.8 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 99.6 99.6 99.6 99.6 99.6 |
-windows-1250 34548 | 99.2 99.2 99.3 99.2 99.2 | 99.2 99.2 99.3 99.2 99.2 | 99.2 99.2 99.3 99.2 99.2 | 99.2 99.2 99.3 99.2 99.2 | 80.1 85.5 99.9 84.5 84.5 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1251 36852 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 99.5 99.5 99.6 99.5 99.5 | 98.9 98.9 99.6 98.9 98.9 | 93.2 93.2 93.2 93.2 93.2 |
-windows-1252 25975 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 99.7 99.7 99.7 99.7 99.7 | 46.5 94.4 99.8 94.4 94.4 | 0.0 98.9 98.9 50.8 98.1 |
-windows-1253 36845 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 33.7 99.8 99.9 93.8 93.9 | 1.0 99.7 99.7 61.1 90.4 |
-windows-1254 36705 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 60.3 99.3 100.0 99.3 99.3 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1255 36774 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 38.0 81.7 94.2 76.3 81.7 | 99.9 99.9 99.9 99.9 99.9 |
-windows-1256 41912 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 52.0 91.6 99.4 52.0 52.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-1257 35316 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.1 |
-windows-1258 36885 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-windows-874 36809 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 100.0 100.0 100.0 100.0 100.0 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 70.6 99.9 |
-x-EUC-TW 26788 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 99.9 99.9 99.9 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 81.1 81.1 81.1 81.1 81.1 |
-x-MacRoman 36756 | 99.6 99.6 99.9 99.6 99.6 | 99.7 99.7 99.9 99.7 99.7 | 99.7 99.7 99.9 99.7 99.7 | 99.7 99.7 99.9 99.7 99.7 | 0.0 0.0 0.0 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 |
-x-mac-cyrillic 36631 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 99.9 99.9 100.0 99.9 99.9 | 0.0 0.0 0.0 0.0 0.0 | 86.7 86.7 86.7 86.7 86.7 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-OVERALL 1469647 | 81.9 84.2 84.3 86.3 86.7 | 95.0 97.2 97.3 99.4 99.8 | 95.0 97.3 97.3 99.4 99.8 | 95.0 97.3 97.3 99.4 99.8 | 45.5 67.0 72.6 54.8 55.3 | 33.5 42.0 42.0 41.2 43.5 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft | T3%=top-3 hit | D%=decode-match | A%=alpha-match
- µs/sample | 18.4 | 13.7 | 12.8 | 13.1 | 146.6 | 16.4 |
-
---- Confusion (All/ML+rules, full, top errors per charset) ---
- Big5-HKSCS 0.0% wrong → GB18030:0.0%, windows-1252:0.0%, ISO-8859-16:0.0%
- EUC-JP 0.2% wrong → GB18030:0.2%, ISO-2022-JP:0.0%, IBM852:0.0%, IBM850:0.0%, EUC-KR:0.0%
- EUC-KR 0.1% wrong → windows-1252:0.0%, GB18030:0.0%
- GB18030 0.0% wrong → windows-1252:0.0%, Big5-HKSCS:0.0%, x-EUC-TW:0.0%, windows-874:0.0%, IBM424-ltr:0.0%
- IBM1047 84.8% wrong → IBM500:84.5%, IBM420-ltr:0.1%, UTF-16-BE:0.1%, IBM424-rtl:0.0%, UTF-16-LE:0.0%
- IBM420-ltr 0.0% wrong → IBM420-rtl:0.0%, IBM500:0.0%
- IBM420-rtl 0.1% wrong → IBM424-rtl:0.0%, IBM420-ltr:0.0%
- IBM424-ltr 0.1% wrong → IBM500:0.1%, IBM424-rtl:0.0%, UTF-8:0.0%, IBM420-ltr:0.0%, x-mac-cyrillic:0.0%
- IBM424-rtl 0.0% wrong → IBM424-ltr:0.0%, UTF-8:0.0%, IBM420-rtl:0.0%
- IBM500 10.9% wrong → IBM1047:10.7%, IBM420-ltr:0.1%, UTF-16-BE:0.1%, IBM424-rtl:0.0%, IBM424-ltr:0.0%
- IBM850 0.3% wrong → windows-1252:0.1%, x-MacRoman:0.1%, IBM852:0.1%, windows-1257:0.0%, ISO-8859-16:0.0%
- IBM852 0.1% wrong → ISO-8859-16:0.0%, windows-1250:0.0%, IBM850:0.0%, windows-1252:0.0%, IBM424-rtl:0.0%
- IBM855 0.0% wrong → windows-1252:0.0%, IBM850:0.0%, x-MacRoman:0.0%, IBM852:0.0%, UTF-8:0.0%
- IBM866 0.0% wrong → x-MacRoman:0.0%, IBM852:0.0%, x-mac-cyrillic:0.0%
- ISO-2022-CN 0.3% wrong → windows-1252:0.3%, ISO-2022-JP:0.0%
- ISO-2022-JP 0.1% wrong → windows-1252:0.1%
- ISO-2022-KR 0.0% wrong → windows-1252:0.0%, ISO-2022-JP:0.0%
- ISO-8859-16 4.6% wrong → windows-1250:4.5%, windows-1252:0.0%, windows-1257:0.0%, IBM852:0.0%, x-MacRoman:0.0%
- ISO-8859-3 0.0% wrong → windows-1252:0.0%, x-MacRoman:0.0%, IBM852:0.0%, UTF-8:0.0%, windows-1254:0.0%
- KOI8-R 0.0% wrong → KOI8-U:0.0%, windows-874:0.0%, windows-1252:0.0%
- KOI8-U 0.3% wrong → KOI8-R:0.3%, windows-1250:0.0%, x-MacRoman:0.0%, windows-1251:0.0%, windows-1252:0.0%
- Shift_JIS 0.0% wrong → IBM852:0.0%, x-mac-cyrillic:0.0%, IBM866:0.0%, GB18030:0.0%, UTF-16-LE:0.0%
- US-ASCII 100.0% wrong → windows-1252:100.0%, x-MacRoman:0.0%, windows-1255:0.0%, ISO-2022-JP:0.0%
- UTF-16-BE 1.2% wrong → windows-1252:1.0%, UTF-16LE:0.1%, ISO-8859-16:0.1%, UTF-8:0.0%, UTF-16-LE:0.0%
- UTF-16-LE 0.6% wrong → windows-1252:0.5%, UTF-16BE:0.0%, IBM852:0.0%, IBM866:0.0%, windows-1256:0.0%
- windows-1250 0.8% wrong → ISO-8859-16:0.7%, IBM852:0.0%, windows-1252:0.0%, x-MacRoman:0.0%, windows-1257:0.0%
- windows-1251 0.5% wrong → x-mac-cyrillic:0.5%, IBM852:0.0%, windows-1250:0.0%, KOI8-R:0.0%
- windows-1252 0.3% wrong → EUC-KR:0.1%, windows-1256:0.0%, IBM850:0.0%, windows-1253:0.0%, KOI8-R:0.0%
- windows-1253 0.1% wrong → windows-1252:0.0%, windows-1255:0.0%, KOI8-R:0.0%, IBM424-ltr:0.0%, IBM852:0.0%
- windows-1254 0.0% wrong → windows-1252:0.0%, windows-1250:0.0%, x-MacRoman:0.0%, IBM852:0.0%, ISO-8859-16:0.0%
- windows-1256 0.0% wrong → windows-1252:0.0%, x-MacRoman:0.0%, windows-874:0.0%, windows-1250:0.0%
- windows-1257 0.1% wrong → windows-1252:0.1%, windows-1250:0.0%, IBM852:0.0%, x-MacRoman:0.0%, IBM850:0.0%
- windows-1258 0.0% wrong → x-MacRoman:0.0%, ISO-8859-3:0.0%, windows-874:0.0%, windows-1257:0.0%
- windows-874 0.0% wrong → windows-1252:0.0%, GB18030:0.0%, IBM850:0.0%, IBM852:0.0%
- x-EUC-TW 0.1% wrong → GB18030:0.1%
- x-MacRoman 0.3% wrong → IBM850:0.2%, windows-1252:0.1%, IBM852:0.0%, ISO-8859-16:0.0%, windows-1257:0.0%
- x-mac-cyrillic 0.1% wrong → windows-1251:0.0%, windows-1252:0.0%, x-MacRoman:0.0%, windows-1250:0.0%
-
-=== Accuracy by probe length (All detector) ===
- Length Strict% Soft% Top3% Decode% Alpha%
- ----------------------------------------------------------
- 8B 59.1 62.6 70.2 83.2 83.4
- 32B 80.8 83.6 86.3 93.4 93.5
- 128B 91.4 93.8 94.2 97.4 97.5
- full 95.0 97.3 97.3 99.4 99.8
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-eval.txt b/docs/modules/ROOT/pages/advanced/charset-detection-eval.txt
deleted file mode 100644
index d8177f2d86..0000000000
--- a/docs/modules/ROOT/pages/advanced/charset-detection-eval.txt
+++ /dev/null
@@ -1,274 +0,0 @@
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-=== Probe length: 20B ===
- N | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
-Charset | Stat R% S% | +ISO R% S% | +CJK R% S% | All R% S% | ICU4J R% S% | juniv R% S% |
-----------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 5000 | 98.0 98.0 | 97.9 97.9 | 97.9 97.9 | 97.9 97.9 | 0.0 9.8 | 0.0 37.9 |
-EUC-JP 5000 | 54.5 54.5 | 54.6 54.6 | 54.7 54.7 | 54.7 54.7 | 0.0 0.0 | 69.7 69.7 |
-EUC-KR 5000 | 66.4 66.4 | 67.6 67.6 | 67.6 67.6 | 67.6 67.6 | 0.0 0.0 | 78.3 78.3 |
-GB18030 5000 | 29.8 29.8 | 30.7 30.7 | 30.8 30.8 | 30.8 30.8 | 0.1 0.1 | 40.6 40.6 |
-IBM1047 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 86.8 | 0.0 0.0 |
-IBM420-ltr 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 80.9 | 0.0 0.0 |
-IBM420-rtl 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 84.2 | 0.0 0.0 |
-IBM424-ltr 6195 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 69.1 | 0.0 0.0 |
-IBM424-rtl 4717 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 69.6 | 0.0 0.0 |
-IBM437 7516 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM500 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 87.7 87.7 | 0.0 0.0 |
-IBM850 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM852 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM855 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 90.7 90.7 |
-IBM866 8442 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 43.0 43.0 | 93.2 93.2 |
-ISO-2022-CN 5000 | 0.0 0.0 | 94.6 94.6 | 94.6 94.6 | 94.6 94.6 | 87.7 87.7 | 0.0 0.0 |
-ISO-2022-JP 5000 | 0.0 0.0 | 94.0 94.0 | 94.0 94.0 | 94.0 94.0 | 77.9 77.9 | 93.9 93.9 |
-ISO-2022-KR 5000 | 0.0 0.0 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 92.6 92.6 | 94.4 94.4 |
-ISO-8859-16 10000 | 38.3 38.3 | 31.6 31.6 | 31.7 31.7 | 31.7 31.7 | 0.0 0.0 | 0.0 0.0 |
-ISO-8859-3 5195 | 10.0 10.0 | 10.3 10.3 | 10.5 10.5 | 10.5 10.5 | 0.0 0.0 | 0.0 0.0 |
-KOI8-R 8411 | 57.0 66.6 | 57.0 66.6 | 57.1 66.7 | 57.1 66.7 | 59.7 59.7 | 94.6 94.6 |
-KOI8-U 5921 | 71.2 79.3 | 71.3 79.5 | 71.3 79.6 | 71.3 79.6 | 0.0 56.6 | 0.0 95.4 |
-Shift_JIS 5000 | 84.3 84.3 | 86.6 86.6 | 86.6 86.6 | 86.6 86.6 | 0.0 0.0 | 71.6 71.6 |
-US-ASCII 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 2.3 | 0.0 0.0 |
-UTF-16-BE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 68.7 68.7 | 0.0 0.0 |
-UTF-16-LE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 69.4 69.4 | 0.0 0.0 |
-UTF-32-BE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 100.0 100.0 | 0.0 0.0 |
-UTF-32-LE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 100.0 100.0 | 0.0 0.0 |
-UTF-8 5000 | 61.4 61.4 | 61.1 61.1 | 61.5 61.5 | 61.5 61.5 | 77.6 77.6 | 78.2 78.2 |
-windows-1250 10000 | 4.7 4.7 | 4.8 4.8 | 4.9 4.9 | 4.9 4.9 | 9.3 48.3 | 0.0 0.0 |
-windows-1251 10000 | 82.4 82.4 | 82.5 82.5 | 82.6 82.6 | 82.6 82.6 | 56.1 56.4 | 71.4 71.5 |
-windows-1252 10000 | 2.6 2.6 | 50.0 50.0 | 50.1 50.1 | 50.1 50.1 | 4.5 70.0 | 0.0 98.7 |
-windows-1253 10000 | 55.2 55.2 | 55.4 55.4 | 55.5 55.5 | 55.5 55.5 | 1.9 63.3 | 0.1 80.8 |
-windows-1254 10000 | 35.8 35.8 | 35.8 35.8 | 35.9 35.9 | 35.9 35.9 | 4.9 52.1 | 0.0 0.0 |
-windows-1255 10000 | 86.7 86.7 | 86.7 86.7 | 86.7 86.7 | 86.7 86.7 | 3.5 27.2 | 93.9 96.2 |
-windows-1256 10000 | 94.1 94.1 | 94.2 94.2 | 94.2 94.2 | 94.2 94.2 | 70.2 78.6 | 0.0 0.0 |
-windows-1257 10000 | 18.8 18.8 | 17.0 17.0 | 17.1 17.1 | 17.1 17.1 | 0.0 0.0 | 0.0 0.0 |
-windows-1258 10000 | 63.0 63.0 | 63.2 63.2 | 63.3 63.3 | 63.3 63.3 | 0.0 0.0 | 0.0 0.0 |
-windows-874 10000 | 79.3 79.3 | 80.6 80.6 | 81.4 81.4 | 81.4 81.4 | 0.0 0.0 | 0.0 0.0 |
-x-EUC-TW 5000 | 99.4 99.4 | 98.3 98.3 | 98.3 98.3 | 98.3 98.3 | 0.0 0.0 | 38.7 38.7 |
-x-ISO-2022-CN-CNS 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-x-MacRoman 10000 | 6.1 6.1 | 6.2 6.2 | 6.3 6.3 | 6.3 6.3 | 0.0 0.0 | 0.0 0.0 |
-x-mac-cyrillic 10000 | 68.8 68.8 | 69.2 69.2 | 69.3 69.3 | 69.3 69.3 | 0.0 0.0 | 49.3 49.3 |
-------------------------------------------------------------------------------------------------------------------------
-OVERALL 326397 | 30.0 30.4 | 35.6 36.0 | 35.7 36.1 | 35.7 36.1 | 20.3 39.0 | 22.9 30.7 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
- µs/sample | 10.0 | 7.3 | 7.4 | 7.0 | 11.9 | 3.3 |
-
-=== Probe length: 50B ===
- N | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
-Charset | Stat R% S% | +ISO R% S% | +CJK R% S% | All R% S% | ICU4J R% S% | juniv R% S% |
-----------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 5000 | 99.4 99.4 | 99.3 99.3 | 99.3 99.3 | 99.3 99.3 | 0.0 95.9 | 0.0 69.1 |
-EUC-JP 5000 | 88.9 88.9 | 89.0 89.0 | 89.0 89.0 | 89.0 89.0 | 86.7 86.7 | 90.2 90.2 |
-EUC-KR 5000 | 92.7 92.7 | 93.3 93.3 | 93.3 93.3 | 93.3 93.3 | 89.4 89.4 | 96.0 96.0 |
-GB18030 5000 | 65.0 65.0 | 66.4 66.4 | 66.5 66.5 | 66.5 66.5 | 71.8 71.8 | 83.2 83.2 |
-IBM1047 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 80.2 | 0.0 0.0 |
-IBM420-ltr 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 93.1 | 0.0 0.0 |
-IBM420-rtl 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 94.7 | 0.0 0.0 |
-IBM424-ltr 6195 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 86.1 | 0.0 0.0 |
-IBM424-rtl 4717 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 87.6 | 0.0 0.0 |
-IBM437 7516 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM500 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 80.2 80.2 | 0.0 0.0 |
-IBM850 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM852 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-IBM855 10000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 97.0 97.0 |
-IBM866 8442 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 62.0 62.0 | 98.1 98.1 |
-ISO-2022-CN 5000 | 0.0 0.0 | 98.6 98.6 | 98.6 98.6 | 98.6 98.6 | 98.5 98.5 | 0.0 0.0 |
-ISO-2022-JP 5000 | 0.0 0.0 | 97.4 97.4 | 97.4 97.4 | 97.4 97.4 | 93.9 93.9 | 97.3 97.3 |
-ISO-2022-KR 5000 | 0.0 0.0 | 98.9 98.9 | 98.9 98.9 | 98.9 98.9 | 98.8 98.8 | 98.9 98.9 |
-ISO-8859-16 10000 | 89.3 89.3 | 80.5 80.5 | 80.7 80.7 | 80.7 80.7 | 0.0 0.0 | 0.0 0.0 |
-ISO-8859-3 5195 | 50.8 50.8 | 49.8 49.8 | 49.9 49.9 | 49.9 49.9 | 0.0 0.0 | 0.0 0.0 |
-KOI8-R 8411 | 86.2 93.6 | 86.2 93.7 | 86.2 93.7 | 86.2 93.7 | 77.6 77.6 | 98.6 98.6 |
-KOI8-U 5921 | 92.7 97.4 | 92.7 97.4 | 92.7 97.5 | 92.7 97.5 | 0.0 74.8 | 0.0 98.2 |
-Shift_JIS 5000 | 95.3 95.3 | 96.1 96.1 | 96.1 96.1 | 96.1 96.1 | 88.5 88.5 | 94.5 94.5 |
-US-ASCII 5000 | 0.0 0.5 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.3 | 0.0 0.0 |
-UTF-16-BE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 85.7 85.7 | 0.0 0.0 |
-UTF-16-LE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 87.6 87.6 | 0.0 0.0 |
-UTF-32-BE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 100.0 100.0 | 0.0 0.0 |
-UTF-32-LE 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 100.0 100.0 | 0.0 0.0 |
-UTF-8 5000 | 84.4 84.4 | 88.3 88.3 | 88.5 88.5 | 88.5 88.5 | 92.4 92.4 | 93.3 93.3 |
-windows-1250 10000 | 31.3 31.3 | 32.3 32.3 | 32.9 32.9 | 32.9 32.9 | 22.3 56.9 | 0.0 0.0 |
-windows-1251 10000 | 95.8 95.8 | 95.8 95.8 | 95.9 95.9 | 95.9 95.9 | 76.0 76.1 | 80.9 80.9 |
-windows-1252 10000 | 20.8 20.8 | 39.1 39.1 | 39.4 39.4 | 39.4 39.4 | 12.8 86.6 | 0.0 99.1 |
-windows-1253 10000 | 90.2 90.2 | 90.3 90.3 | 90.3 90.3 | 90.3 90.3 | 8.2 86.8 | 0.1 91.5 |
-windows-1254 10000 | 80.1 80.1 | 80.0 80.0 | 80.0 80.0 | 80.0 80.0 | 12.3 73.1 | 0.0 0.0 |
-windows-1255 10000 | 97.8 97.8 | 97.8 97.8 | 97.8 97.8 | 97.8 97.8 | 9.4 40.2 | 98.1 99.4 |
-windows-1256 10000 | 99.1 99.1 | 99.2 99.2 | 99.2 99.2 | 99.2 99.2 | 88.1 91.5 | 0.0 0.0 |
-windows-1257 10000 | 72.0 72.0 | 67.2 67.2 | 67.3 67.3 | 67.3 67.3 | 0.0 0.0 | 0.0 0.0 |
-windows-1258 10000 | 96.1 96.1 | 96.2 96.2 | 96.3 96.3 | 96.3 96.3 | 0.0 0.0 | 0.0 0.0 |
-windows-874 10000 | 95.3 95.3 | 95.8 95.8 | 96.0 96.0 | 96.0 96.0 | 0.0 0.0 | 0.0 0.0 |
-x-EUC-TW 5000 | 99.5 99.5 | 99.3 99.3 | 99.3 99.3 | 99.3 99.3 | 0.0 0.0 | 67.1 67.1 |
-x-ISO-2022-CN-CNS 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-x-MacRoman 10000 | 58.5 58.5 | 56.4 56.4 | 56.6 56.6 | 56.6 56.6 | 0.0 0.0 | 0.0 0.0 |
-x-mac-cyrillic 10000 | 81.0 81.0 | 81.2 81.2 | 81.2 81.2 | 81.2 81.2 | 0.0 0.0 | 62.3 62.3 |
-------------------------------------------------------------------------------------------------------------------------
-OVERALL 326397 | 45.2 45.4 | 49.9 50.2 | 49.9 50.2 | 49.9 50.2 | 29.8 52.4 | 26.5 35.2 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
- µs/sample | 10.7 | 8.2 | 8.1 | 8.1 | 21.6 | 4.1 |
-
-=== Probe length: 100B ===
- N | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
-Charset | Stat R% S% | +ISO R% S% | +CJK R% S% | All R% S% | ICU4J R% S% | juniv R% S% |
-----------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 5000 | 99.7 99.7 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 0.0 97.6 | 0.0 69.7 |
-EUC-JP 5000 | 95.8 95.8 | 95.8 95.8 | 95.8 95.8 | 95.8 95.8 | 97.0 97.0 | 95.7 95.7 |
-EUC-KR 5000 | 95.3 95.3 | 95.4 95.4 | 95.4 95.4 | 95.4 95.4 | 98.4 98.4 | 99.1 99.1 |
-GB18030 5000 | 85.1 85.1 | 85.9 85.9 | 85.9 85.9 | 85.9 85.9 | 95.9 95.9 | 97.9 97.9 |
-IBM1047 10000 | 49.5 94.7 | 49.6 94.9 | 49.6 94.9 | 49.6 94.9 | 0.0 83.2 | 0.0 0.0 |
-IBM420-ltr 10000 | 92.6 95.0 | 92.6 95.0 | 92.6 95.0 | 92.6 95.0 | 0.0 95.0 | 0.0 0.0 |
-IBM420-rtl 10000 | 94.1 94.6 | 94.1 94.6 | 94.1 94.6 | 94.1 94.6 | 0.0 96.3 | 0.0 0.0 |
-IBM424-ltr 6195 | 91.6 91.8 | 74.8 75.0 | 74.8 75.0 | 74.8 75.0 | 0.0 92.4 | 0.0 0.0 |
-IBM424-rtl 4717 | 93.0 95.1 | 74.2 75.5 | 74.2 75.5 | 74.2 75.5 | 0.0 89.4 | 0.0 0.0 |
-IBM437 7516 | 0.0 79.0 | 0.0 77.8 | 0.0 77.8 | 0.0 77.8 | 0.0 0.0 | 0.0 0.0 |
-IBM500 10000 | 56.0 94.9 | 56.1 95.1 | 56.1 95.1 | 56.1 95.1 | 83.5 83.5 | 0.0 0.0 |
-IBM850 10000 | 79.8 79.8 | 79.5 79.5 | 79.5 79.5 | 79.5 79.5 | 0.0 0.0 | 0.0 0.0 |
-IBM852 10000 | 77.4 77.4 | 78.1 78.1 | 78.1 78.1 | 78.1 78.1 | 0.0 0.0 | 0.0 0.0 |
-IBM855 10000 | 94.6 94.6 | 94.7 94.7 | 94.7 94.7 | 94.7 94.7 | 0.0 0.0 | 99.0 99.0 |
-IBM866 8442 | 95.9 95.9 | 96.0 96.0 | 96.0 96.0 | 96.0 96.0 | 78.8 78.8 | 99.3 99.3 |
-ISO-2022-CN 5000 | 0.0 0.0 | 99.1 99.1 | 99.1 99.1 | 99.1 99.1 | 99.0 99.0 | 0.0 0.0 |
-ISO-2022-JP 5000 | 0.0 0.0 | 99.1 99.1 | 99.1 99.1 | 99.1 99.1 | 98.8 98.8 | 99.1 99.1 |
-ISO-2022-KR 5000 | 0.0 0.0 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 |
-ISO-8859-16 10000 | 92.0 92.0 | 89.1 89.1 | 89.1 89.1 | 89.1 89.1 | 0.0 0.0 | 0.0 0.0 |
-ISO-8859-3 5195 | 78.4 78.4 | 78.5 78.5 | 78.5 78.5 | 78.5 78.5 | 0.0 0.0 | 0.0 0.0 |
-KOI8-R 8411 | 95.7 99.0 | 95.7 99.0 | 95.7 99.0 | 95.7 99.0 | 90.5 90.5 | 99.4 99.4 |
-KOI8-U 5921 | 96.9 99.5 | 96.9 99.5 | 96.9 99.5 | 96.9 99.5 | 0.0 85.4 | 0.0 98.9 |
-Shift_JIS 5000 | 97.7 97.7 | 98.0 98.0 | 98.0 98.0 | 98.0 98.0 | 97.8 97.8 | 98.7 98.7 |
-US-ASCII 5000 | 0.0 0.8 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-UTF-16-BE 5000 | 94.0 94.1 | 93.9 93.9 | 93.9 93.9 | 93.9 93.9 | 85.5 85.5 | 0.0 0.0 |
-UTF-16-LE 5000 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 87.0 87.0 | 0.0 0.0 |
-UTF-32-BE 5000 | 94.9 94.9 | 94.7 94.7 | 94.7 94.7 | 94.7 94.7 | 100.0 100.0 | 0.0 0.0 |
-UTF-32-LE 5000 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 100.0 100.0 | 0.0 0.0 |
-UTF-8 5000 | 90.6 90.6 | 93.2 93.2 | 93.2 93.2 | 93.2 93.2 | 97.3 97.3 | 97.2 97.2 |
-windows-1250 10000 | 55.3 55.3 | 56.4 56.4 | 56.5 56.5 | 56.5 56.5 | 34.4 62.5 | 0.0 0.0 |
-windows-1251 10000 | 98.8 98.8 | 98.8 98.8 | 98.8 98.8 | 98.8 98.8 | 85.7 85.7 | 86.2 86.2 |
-windows-1252 10000 | 34.4 34.4 | 40.0 40.0 | 40.4 40.4 | 40.4 40.4 | 21.3 91.4 | 0.0 99.4 |
-windows-1253 10000 | 97.1 97.1 | 97.1 97.1 | 97.1 97.1 | 97.1 97.1 | 15.4 93.7 | 0.2 95.3 |
-windows-1254 10000 | 93.1 93.1 | 93.0 93.0 | 93.0 93.0 | 93.0 93.0 | 21.2 83.8 | 0.0 0.0 |
-windows-1255 10000 | 99.0 99.0 | 99.0 99.0 | 99.0 99.0 | 99.0 99.0 | 14.6 46.8 | 99.0 99.9 |
-windows-1256 10000 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 94.6 96.1 | 0.0 0.0 |
-windows-1257 10000 | 87.1 87.1 | 85.7 85.7 | 85.7 85.7 | 85.7 85.7 | 0.0 0.0 | 0.0 0.0 |
-windows-1258 10000 | 97.0 97.0 | 97.0 97.0 | 97.0 97.0 | 97.0 97.0 | 0.0 0.0 | 0.0 0.0 |
-windows-874 10000 | 97.7 97.7 | 97.7 97.7 | 97.7 97.7 | 97.7 97.7 | 0.0 0.0 | 0.0 0.0 |
-x-EUC-TW 5000 | 99.7 99.7 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 0.0 0.0 | 67.7 67.7 |
-x-ISO-2022-CN-CNS 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-x-MacRoman 10000 | 80.0 80.0 | 79.2 79.2 | 79.2 79.2 | 79.2 79.2 | 0.0 0.0 | 0.0 0.0 |
-x-mac-cyrillic 10000 | 88.6 88.6 | 88.7 88.7 | 88.7 88.7 | 88.7 88.7 | 0.0 0.0 | 70.1 70.1 |
-------------------------------------------------------------------------------------------------------------------------
-OVERALL 326397 | 78.0 82.6 | 82.1 86.7 | 82.1 86.7 | 82.1 86.7 | 33.4 56.3 | 27.6 36.4 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
- µs/sample | 10.8 | 8.2 | 8.0 | 7.9 | 36.3 | 5.7 |
-
-=== Probe length: 200B ===
- N | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
-Charset | Stat R% S% | +ISO R% S% | +CJK R% S% | All R% S% | ICU4J R% S% | juniv R% S% |
-----------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 5000 | 99.8 99.8 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 0.0 97.8 | 0.0 69.7 |
-EUC-JP 5000 | 98.4 98.4 | 98.4 98.4 | 98.4 98.4 | 98.4 98.4 | 99.2 99.2 | 97.9 97.9 |
-EUC-KR 5000 | 99.1 99.1 | 99.1 99.1 | 99.1 99.1 | 99.1 99.1 | 99.8 99.8 | 99.8 99.8 |
-GB18030 5000 | 94.0 94.0 | 94.5 94.5 | 94.5 94.5 | 94.5 94.5 | 98.4 98.4 | 99.4 99.4 |
-IBM1047 10000 | 52.7 95.7 | 52.8 95.8 | 52.8 95.8 | 52.8 95.8 | 0.0 86.8 | 0.0 0.0 |
-IBM420-ltr 10000 | 95.3 96.1 | 95.3 96.1 | 95.3 96.1 | 95.3 96.1 | 0.0 96.8 | 0.0 0.0 |
-IBM420-rtl 10000 | 95.3 95.4 | 95.3 95.4 | 95.3 95.4 | 95.3 95.4 | 0.0 97.3 | 0.0 0.0 |
-IBM424-ltr 6195 | 94.9 94.9 | 87.8 87.8 | 87.8 87.8 | 87.8 87.8 | 0.0 93.4 | 0.0 0.0 |
-IBM424-rtl 4717 | 95.6 96.0 | 86.7 87.2 | 86.7 87.2 | 86.7 87.2 | 0.0 87.3 | 0.0 0.0 |
-IBM437 7516 | 0.0 92.1 | 0.0 92.0 | 0.0 92.0 | 0.0 92.0 | 0.0 0.0 | 0.0 0.0 |
-IBM500 10000 | 56.0 96.0 | 56.0 96.0 | 56.0 96.0 | 56.0 96.0 | 86.8 86.8 | 0.0 0.0 |
-IBM850 10000 | 92.2 92.2 | 92.3 92.3 | 92.3 92.3 | 92.3 92.3 | 0.0 0.0 | 0.0 0.0 |
-IBM852 10000 | 90.0 90.0 | 90.7 90.7 | 90.7 90.7 | 90.7 90.7 | 0.0 0.0 | 0.0 0.0 |
-IBM855 10000 | 95.7 95.7 | 95.7 95.7 | 95.7 95.7 | 95.7 95.7 | 0.0 0.0 | 99.8 99.8 |
-IBM866 8442 | 96.2 96.2 | 96.2 96.2 | 96.2 96.2 | 96.2 96.2 | 89.5 89.5 | 99.7 99.7 |
-ISO-2022-CN 5000 | 0.0 0.0 | 99.5 99.5 | 99.5 99.5 | 99.5 99.5 | 99.4 99.4 | 0.0 0.0 |
-ISO-2022-JP 5000 | 0.0 0.0 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.4 99.4 | 99.6 99.6 |
-ISO-2022-KR 5000 | 0.0 0.0 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 |
-ISO-8859-16 10000 | 95.3 95.3 | 94.9 94.9 | 94.9 94.9 | 94.9 94.9 | 0.0 0.0 | 0.0 0.0 |
-ISO-8859-3 5195 | 93.9 93.9 | 94.3 94.3 | 94.3 94.3 | 94.3 94.3 | 0.0 0.0 | 0.0 0.0 |
-KOI8-R 8411 | 98.6 99.8 | 98.6 99.8 | 98.6 99.8 | 98.6 99.8 | 95.8 95.8 | 99.7 99.7 |
-KOI8-U 5921 | 98.9 99.9 | 98.9 99.9 | 98.9 99.9 | 98.9 99.9 | 0.0 92.5 | 0.0 99.3 |
-Shift_JIS 5000 | 99.3 99.3 | 99.4 99.4 | 99.4 99.4 | 99.4 99.4 | 99.7 99.7 | 99.7 99.7 |
-US-ASCII 5000 | 0.0 0.5 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-UTF-16-BE 5000 | 94.5 94.5 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 84.9 84.9 | 0.0 0.0 |
-UTF-16-LE 5000 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 94.4 94.4 | 86.8 86.8 | 0.0 0.0 |
-UTF-32-BE 5000 | 95.2 95.2 | 95.0 95.0 | 95.0 95.0 | 95.0 95.0 | 100.0 100.0 | 0.0 0.0 |
-UTF-32-LE 5000 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 100.0 100.0 | 0.0 0.0 |
-UTF-8 5000 | 96.1 96.1 | 96.7 96.7 | 96.7 96.7 | 96.7 96.7 | 99.5 99.5 | 98.1 98.1 |
-windows-1250 10000 | 73.6 73.6 | 74.3 74.3 | 74.4 74.4 | 74.4 74.4 | 49.3 70.3 | 0.0 0.0 |
-windows-1251 10000 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 92.6 92.6 | 89.1 89.1 |
-windows-1252 10000 | 47.7 47.7 | 48.5 48.5 | 48.7 48.7 | 48.7 48.7 | 28.5 94.0 | 0.0 99.5 |
-windows-1253 10000 | 99.4 99.4 | 99.4 99.4 | 99.4 99.4 | 99.4 99.4 | 22.0 97.2 | 0.3 97.6 |
-windows-1254 10000 | 98.4 98.4 | 98.4 98.4 | 98.4 98.4 | 98.4 98.4 | 33.4 93.5 | 0.0 0.0 |
-windows-1255 10000 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 20.2 57.9 | 99.7 99.9 |
-windows-1256 10000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 97.1 98.2 | 0.0 0.0 |
-windows-1257 10000 | 95.1 95.1 | 95.1 95.1 | 95.1 95.1 | 95.1 95.1 | 0.0 0.0 | 0.0 0.0 |
-windows-1258 10000 | 99.0 99.0 | 99.0 99.0 | 99.0 99.0 | 99.0 99.0 | 0.0 0.0 | 0.0 0.0 |
-windows-874 10000 | 99.4 99.4 | 99.4 99.4 | 99.4 99.4 | 99.4 99.4 | 0.0 0.0 | 0.0 0.0 |
-x-EUC-TW 5000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 0.0 0.0 | 67.7 67.7 |
-x-ISO-2022-CN-CNS 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-x-MacRoman 10000 | 93.5 93.5 | 93.5 93.5 | 93.5 93.5 | 93.5 93.5 | 0.0 0.0 | 0.0 0.0 |
-x-mac-cyrillic 10000 | 93.5 93.5 | 93.5 93.5 | 93.5 93.5 | 93.5 93.5 | 0.0 0.0 | 74.9 74.9 |
-------------------------------------------------------------------------------------------------------------------------
-OVERALL 326397 | 82.1 86.9 | 86.5 91.2 | 86.5 91.3 | 86.5 91.3 | 35.8 58.6 | 28.0 36.9 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
- µs/sample | 12.4 | 9.2 | 8.7 | 8.9 | 60.7 | 8.2 |
-
-=== Probe length: full ===
- N | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
-Charset | Stat R% S% | +ISO R% S% | +CJK R% S% | All R% S% | ICU4J R% S% | juniv R% S% |
-----------------------------------------------------------------------------------------------------------------------------------
-Big5-HKSCS 5000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 0.0 97.8 | 0.0 69.7 |
-EUC-JP 5000 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.9 99.9 | 99.0 99.0 |
-EUC-KR 5000 | 99.7 99.7 | 99.7 99.7 | 99.7 99.7 | 99.7 99.7 | 100.0 100.0 | 99.9 99.9 |
-GB18030 5000 | 97.1 97.1 | 97.5 97.5 | 97.5 97.5 | 97.5 97.5 | 99.4 99.4 | 99.7 99.7 |
-IBM1047 10000 | 59.1 96.0 | 59.1 96.0 | 59.1 96.0 | 59.1 96.0 | 0.0 90.6 | 0.0 0.0 |
-IBM420-ltr 10000 | 96.0 96.2 | 96.0 96.2 | 96.0 96.2 | 96.0 96.2 | 0.0 97.8 | 0.0 0.0 |
-IBM420-rtl 10000 | 95.6 95.7 | 95.6 95.7 | 95.6 95.7 | 95.6 95.7 | 0.0 98.1 | 0.0 0.0 |
-IBM424-ltr 6195 | 95.6 95.6 | 95.5 95.6 | 95.5 95.6 | 95.5 95.6 | 0.0 93.1 | 0.0 0.0 |
-IBM424-rtl 4717 | 96.2 96.3 | 96.2 96.3 | 96.2 96.3 | 96.2 96.3 | 0.0 87.3 | 0.0 0.0 |
-IBM437 7516 | 0.0 95.9 | 0.0 96.0 | 0.0 96.0 | 0.0 96.0 | 0.0 0.0 | 0.0 0.0 |
-IBM500 10000 | 52.6 96.3 | 52.6 96.3 | 52.6 96.3 | 52.6 96.3 | 90.6 90.6 | 0.0 0.0 |
-IBM850 10000 | 95.8 95.8 | 95.9 95.9 | 95.9 95.9 | 95.9 95.9 | 0.0 0.0 | 0.0 0.0 |
-IBM852 10000 | 95.2 95.2 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 0.0 0.0 | 0.0 0.0 |
-IBM855 10000 | 95.9 95.9 | 95.9 95.9 | 95.9 95.9 | 95.9 95.9 | 0.0 0.0 | 99.9 99.9 |
-IBM866 8442 | 96.2 96.2 | 96.2 96.2 | 96.2 96.2 | 96.2 96.2 | 95.8 95.8 | 99.8 99.8 |
-ISO-2022-CN 5000 | 0.0 0.0 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.7 99.7 | 0.0 0.0 |
-ISO-2022-JP 5000 | 0.0 0.0 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.7 99.7 | 99.8 99.8 |
-ISO-2022-KR 5000 | 0.0 0.0 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 |
-ISO-8859-16 10000 | 98.5 98.5 | 98.5 98.5 | 98.5 98.5 | 98.5 98.5 | 0.0 0.0 | 0.0 0.0 |
-ISO-8859-3 5195 | 98.4 98.4 | 98.7 98.7 | 98.7 98.7 | 98.7 98.7 | 0.0 0.0 | 0.0 0.0 |
-KOI8-R 8411 | 99.4 99.8 | 99.4 99.8 | 99.4 99.8 | 99.4 99.8 | 98.0 98.0 | 99.8 99.8 |
-KOI8-U 5921 | 99.5 99.9 | 99.5 99.9 | 99.5 99.9 | 99.5 99.9 | 0.0 96.5 | 0.0 99.8 |
-Shift_JIS 5000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 100.0 100.0 | 99.9 99.9 |
-US-ASCII 5000 | 0.0 0.2 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-UTF-16-BE 5000 | 94.7 94.7 | 94.7 94.7 | 94.7 94.7 | 94.7 94.7 | 84.6 84.6 | 0.0 0.0 |
-UTF-16-LE 5000 | 94.5 94.5 | 94.5 94.5 | 94.5 94.5 | 94.5 94.5 | 86.6 86.6 | 0.0 0.0 |
-UTF-32-BE 5000 | 95.2 95.2 | 95.2 95.2 | 95.2 95.2 | 95.2 95.2 | 100.0 100.0 | 0.0 0.0 |
-UTF-32-LE 5000 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 95.5 95.5 | 100.0 100.0 | 0.0 0.0 |
-UTF-8 5000 | 98.8 98.8 | 98.0 98.0 | 98.1 98.1 | 98.1 98.1 | 99.9 99.9 | 97.7 97.7 |
-windows-1250 10000 | 83.5 83.5 | 83.7 83.7 | 83.8 83.8 | 83.8 83.8 | 65.4 78.8 | 0.0 0.0 |
-windows-1251 10000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 96.4 96.4 | 90.7 90.7 |
-windows-1252 10000 | 63.8 63.8 | 64.0 64.0 | 64.0 64.0 | 64.0 64.0 | 41.4 96.2 | 0.0 99.6 |
-windows-1253 10000 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 33.1 98.9 | 0.4 99.0 |
-windows-1254 10000 | 99.3 99.3 | 99.3 99.3 | 99.3 99.3 | 99.3 99.3 | 48.7 97.6 | 0.0 0.0 |
-windows-1255 10000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 30.4 71.6 | 99.9 100.0 |
-windows-1256 10000 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 99.9 99.9 | 98.1 99.0 | 0.0 0.0 |
-windows-1257 10000 | 98.5 98.5 | 98.6 98.6 | 98.6 98.6 | 98.6 98.6 | 0.0 0.0 | 0.0 0.0 |
-windows-1258 10000 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 99.6 99.6 | 0.0 0.0 | 0.0 0.0 |
-windows-874 10000 | 99.7 99.7 | 99.8 99.8 | 99.8 99.8 | 99.8 99.8 | 0.0 0.0 | 0.0 0.0 |
-x-EUC-TW 5000 | 99.9 99.9 | 100.0 100.0 | 100.0 100.0 | 100.0 100.0 | 0.0 0.0 | 67.7 67.7 |
-x-ISO-2022-CN-CNS 5000 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
-x-MacRoman 10000 | 98.5 98.5 | 98.6 98.6 | 98.7 98.7 | 98.7 98.7 | 0.0 0.0 | 0.0 0.0 |
-x-mac-cyrillic 10000 | 96.9 96.9 | 96.9 96.9 | 96.9 96.9 | 96.9 96.9 | 0.0 0.0 | 75.0 75.0 |
-------------------------------------------------------------------------------------------------------------------------
-OVERALL 326397 | 84.1 88.8 | 88.7 93.4 | 88.7 93.4 | 88.7 93.4 | 38.4 60.3 | 28.1 37.0 |
- Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
- µs/sample | 16.4 | 11.5 | 10.8 | 11.4 | 152.6 | 18.0 |
diff --git a/docs/modules/ROOT/pages/advanced/charsoup-supported-languages.adoc b/docs/modules/ROOT/pages/advanced/charsoup-supported-languages.adoc
index 566db0d846..4c57fb60e4 100644
--- a/docs/modules/ROOT/pages/advanced/charsoup-supported-languages.adoc
+++ b/docs/modules/ROOT/pages/advanced/charsoup-supported-languages.adoc
@@ -15,11 +15,14 @@
// limitations under the License.
//
-// Regenerate with DumpLabels tool after model training.
+// Regenerate after model training with PrintDiscLabels (tika-langdetect-charsoup,
+// test scope), passing the .bin path; it prints the model's labels sorted.
= CharSoup Supported Languages
-Language codes are ISO 639-3 (with private-use suffixes for romanized variants).
+Language codes are ISO 639-3, apart from three orthography variants that carry a
+private-use subtag: `cdo-x-rom` and `hak-x-rom` (romanized) and `be-x-old`
+(Belarusian Taraškievica).
== Model 20260320
diff --git a/docs/modules/ROOT/pages/advanced/embedded-documents.adoc b/docs/modules/ROOT/pages/advanced/embedded-documents.adoc
index b8772989dd..fc083ac31e 100644
--- a/docs/modules/ROOT/pages/advanced/embedded-documents.adoc
+++ b/docs/modules/ROOT/pages/advanced/embedded-documents.adoc
@@ -80,12 +80,13 @@ This field may have fewer "unknown" file names than `EMBEDDED_RESOURCE_PATH`.
=== Resource Naming
`TikaCoreProperties.RESOURCE_NAME_KEY` (`tk:resource-name`)::
-The file name (not path) of the resource. Tika makes a best effort to determine a meaningful
+The name a parser recorded for the resource. Tika makes a best effort to determine a meaningful
name from the container's metadata. When unavailable, Tika falls back to synthetic names
-such as `embedded-1.jpeg`.
+such as `embedded-1.jpg`.
+
-NOTE: In Tika 4.x, this field contains only the file name. Use `INTERNAL_PATH` for the
-full path as stored in the container.
+NOTE: The archive parsers (zip, tar, and friends) store the full entry path here — the same
+value as `INTERNAL_PATH`. Most other parsers store a bare file name. Tika strips any path
+when it builds `EMBEDDED_RESOURCE_PATH`.
== Container Metadata (From the File)
@@ -101,7 +102,7 @@ container knows about where the file lives in its internal structure:
+
* In a ZIP: the entry path (e.g., `reports/Q1/sales.xlsx`)
* In a PST: the folder path plus message name (e.g., `Inbox/Important/Meeting notes.msg`)
-* In an OOXML document: the part name (e.g., `xl/media/image1.png`)
+* In an OOXML document: the part name (e.g., `/xl/media/image1.png`)
+
This differs fundamentally from `EMBEDDED_RESOURCE_PATH`:
+
@@ -111,8 +112,8 @@ This differs fundamentally from `EMBEDDED_RESOURCE_PATH`:
`TikaCoreProperties.ORIGINAL_RESOURCE_NAME` (`tk:orig-resource-name`)::
For some file formats, the file path where the document was last saved on the creator's
system. For example, an `.xlsx` file named `budget.xlsx` may include a metadata property
-storing where it was last saved: `C:\Users\Alice\budget.xlsx`. This is not specific to
-embedded files - it's a property that certain file formats preserve about themselves.
+storing where it was last saved: `C:\Users\Alice\budget.xlsx`. Parsers also use it for the
+original file name or path a container recorded for an attachment.
== Microsoft-Specific Metadata
@@ -206,7 +207,7 @@ itself contains an embedded image:
|`reports/Q1/sales.xlsx` (from ZIP entry)
|`RESOURCE_NAME_KEY`
-|`sales.xlsx`
+|`reports/Q1/sales.xlsx` (the ZIP parser stores the entry path)
|`EMBEDDED_RESOURCE_PATH`
|`/sales.xlsx`
@@ -219,7 +220,7 @@ itself contains an embedded image:
|`/1/2` (embedded in file with ID=1)
|`INTERNAL_PATH`
-|`xl/media/image1.png` (from XLSX structure)
+|`/xl/media/image1.png` (from XLSX structure)
|`RESOURCE_NAME_KEY`
|`image1.png`
@@ -244,7 +245,7 @@ container metadata:
* `INTERNAL_PATH` for the spreadsheet (`reports/Q1/sales.xlsx`) is what the _ZIP file knows_
about where that entry was stored - its internal folder structure.
-* `INTERNAL_PATH` for the image (`xl/media/image1.png`) is what the _XLSX file knows_ about
+* `INTERNAL_PATH` for the image (`/xl/media/image1.png`) is what the _XLSX file knows_ about
where that media file lives - its internal OOXML part name.
Notice that `INTERNAL_PATH` resets at each container boundary. The image's internal path
diff --git a/docs/modules/ROOT/pages/advanced/index.adoc b/docs/modules/ROOT/pages/advanced/index.adoc
index abd555de9c..2f1045a417 100644
--- a/docs/modules/ROOT/pages/advanced/index.adoc
+++ b/docs/modules/ROOT/pages/advanced/index.adoc
@@ -26,23 +26,34 @@ xref:pipes/index.adoc[Tika Pipes] (pipeline + Pipes-mode tuning),
xref:using-tika/server/index.adoc[Tika Server] (REST + server config), or
xref:using-tika/cli/index.adoc[Tika CLI] (`tika-app` flags). The
xref:advanced/setting-limits.adoc[Setting Limits] page is the model — it
-covers Java, JSON, and CLI side by side. Filing issues against specific
+covers the Java API and the JSON config side by side. Filing issues against specific
advanced pages where the JSON/CLI equivalent isn't documented yet helps us
prioritize the gap.
-== Topics
+== Running untrusted content safely
-* xref:advanced/language-detection.adoc[Language Detection] - Built-in bigram language detector, training pipeline, and comparison with OpenNLP
-* xref:advanced/language-detection-build.adoc[Building the Language Detector] - Full training log, decisions, and benchmark results vs OpenNLP
-* xref:advanced/robustness.adoc[Robustness] - Process isolation and fault tolerance when parsing untrusted content
-* xref:advanced/setting-limits.adoc[Setting Limits] - Limiting metadata size and resource consumption when processing untrusted content
-* xref:advanced/spooling.adoc[TikaInputStream and Spooling] - Understanding how TikaInputStream handles buffering, caching, and spooling to disk
-* xref:advanced/embedded-documents.adoc[Embedded Document Metadata] - Understanding how Tika tracks embedded documents and their paths
-* xref:advanced/zip-detection.adoc[ZIP Detection and Salvaging] - How Tika detects and recovers truncated ZIP-based files
+* xref:advanced/robustness.adoc[Robustness] — process isolation and fault tolerance
+* xref:advanced/setting-limits.adoc[Setting Limits] — depth, size, timeout, and metadata bounds
+* xref:advanced/spooling.adoc[TikaInputStream and Spooling] — buffering, rewinding, and spilling to disk
-* xref:advanced/local-vlm-server.adoc[Running a Local VLM Server] - Run an open-source VLM locally as an OpenAI-compatible endpoint for air-gapped OCR
+== Detection and text quality
-== Integration Testing
+* xref:advanced/language-detection.adoc[Language Detection] — the built-in CharSoup detector
+* xref:advanced/language-detection-build.adoc[Building the Language Detector] — training and benchmarks
+* xref:advanced/charsoup-supported-languages.adoc[CharSoup Supported Languages] — the 204 output classes
+* xref:advanced/junk-detection.adoc[Text Quality Scoring (Junk Detection)] — scoring text for corruption
+* xref:advanced/junk-detection-build.adoc[Building the Junk Detector] — training and model format
+* xref:advanced/charset-detection-design.adoc[Charset Detection Pipeline] — how the encoding-detector chain arbitrates
+* xref:advanced/zip-detection.adoc[ZIP Detection and Salvaging] — recovering truncated ZIP-based files
-* xref:advanced/integration-testing/tika-app.adoc[Testing with Tika App] - Integration testing strategies for Tika App
-* xref:advanced/integration-testing/tika-server.adoc[Testing with Tika Server] - Integration testing strategies for Tika Server
+== Parsing internals
+
+* xref:advanced/embedded-documents.adoc[Embedded Document Metadata] — how Tika tracks embedded documents and their paths
+* xref:advanced/local-vlm-server.adoc[Running a Local VLM Server] — an OpenAI-compatible endpoint for air-gapped OCR
+
+== Integration testing
+
+* xref:advanced/integration-testing/tika-app.adoc[Testing with Tika App]
+* xref:advanced/integration-testing/tika-server.adoc[Testing with Tika Server]
+* xref:advanced/integration-testing/run-uat-script.adoc[Tika-Server REST UAT Script]
+* xref:advanced/integration-testing/tika-eval-regression.adoc[Regression evaluation with tika-app + tika-eval]
diff --git a/docs/modules/ROOT/pages/advanced/integration-testing/run-uat-script.adoc b/docs/modules/ROOT/pages/advanced/integration-testing/run-uat-script.adoc
index 535d4df1a6..f1d745cf1a 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/run-uat-script.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/run-uat-script.adoc
@@ -31,9 +31,13 @@ release-tools/uat/
└── test-files/
├── testPDF.pdf
├── testHTML.html
+ ├── testOCR_spacing.png
└── test_recursive_embedded.docx
----
+All four files must be present -- the script exits 2 if any is missing. Point
+`TIKA_UAT_TEST_FILES` at another directory to override the location.
+
== What it covers
Roughly 30 REST endpoint checks across the default-mode endpoints, header
@@ -45,7 +49,7 @@ Coverage includes:
* `/version`, `/parsers`, `/detectors`, `/mime-types` (introspection)
* `/detect` (mime detection)
-* `/tika`, `/tika/text`, `/tika/xml`, `/tika/json` (parse)
+* `/tika/text`, `/tika/html`, `/tika/xml`, `/tika/json` (parse)
* `/meta`, `/meta/\{field}` (metadata)
* `/rmeta`, `/rmeta/text` (recursive metadata)
* `/unpack/all` (embedded extraction; verifies the response is a valid zip)
@@ -58,7 +62,8 @@ Coverage includes:
`/unpack/all/config`.
* `/status` is not registered by default (returns 404 unless explicitly listed
under `endpoints`)
-* `X-Tika-OCRskipOcr` header
+* A removed `X-Tika-OCRskipOcr` header is ignored rather than rejected (asserts HTTP 200
+ only; the `X-Tika-OCR*`/`X-Tika-PDF*` families no longer exist)
* Handler selection: `/tika/text` is body-only (no `<title>` in the output),
`/tika/json` with no handler named matches `/tika/json/md` (markdown is the
default), and an unrecognized handler name returns 400 rather than silently
@@ -85,14 +90,14 @@ or stop the server itself.
[source,bash]
----
-release-tools/uat/run-uat.sh [host]
-# default host: http://localhost:9998
+release-tools/uat/run-uat.sh [base-url]
+# default: http://localhost:9998
----
Exit code: `0` on all-pass, `1` on any failure. Failed checks print the
expected pattern and a truncated response body.
-=== Against the unpacked bin.zip distribution
+=== Against the unpacked server distribution zip
[source,bash]
----
@@ -100,13 +105,14 @@ unzip tika-server-standard-<VERSION>.zip -d /tmp/tika-server-dist
cd /tmp/tika-server-dist
java -jar tika-server-standard-<VERSION>.jar -p 9998 -h localhost &
sleep 12
-~/path/to/tika/release-tools/uat/run-uat.sh
+<tika-checkout>/release-tools/uat/run-uat.sh
----
=== Against the Docker image
-The `docker-tool.sh test-uat` subcommand wraps starting the container, waiting
-for `/version`, running the UAT, and stopping the container:
+The `docker-tool.sh test-uat` subcommand wraps starting each container, waiting
+for `/version`, running the UAT, and stopping the container. It runs against both
+the minimal and the `-full` image:
[source,bash]
----
@@ -119,8 +125,11 @@ cd tika-server/docker-build
The Maven module `tika-e2e-tests/tika-server` unpacks the distribution zip, forks
`java -jar tika-server-standard-<VERSION>.jar`, and invokes this script via
`org.apache.tika.server.e2e.RunUatSmokeTest`. The CI workflow
-`.github/workflows/main-jdk17-build.yml` runs this automatically on every PR
-via `mvn -pl tika-e2e-tests -am clean verify -Pe2e`.
+`.github/workflows/main-jdk17-build.yml` runs it on pull requests to `main` (its
+`e2e-tests` job installs the full reactor, then runs
+`mvn -pl :tika-e2e-tests-server,:tika-grpc-e2e-test clean verify -Pe2e`). `-pl` must
+name the leaf modules: `tika-e2e-tests` is an aggregator pom, and Maven does not pull
+in a selected aggregator's children.
== When to use it
@@ -130,7 +139,7 @@ via `mvn -pl tika-e2e-tests -am clean verify -Pe2e`.
* *Pre-publish docker verification.* Run via `docker-tool.sh test-uat` after
building a new image and before tagging it for release.
* *Local development sanity check.* When changing anything in
- `tika-server-core` or the bin.zip assembly descriptor, run the UAT against
+ `tika-server-core` or the server assembly descriptor, run the UAT against
the build output to confirm you didn't regress endpoint behavior.
* *Adding new endpoints.* When a new REST endpoint lands, add a corresponding
check to the script so future regressions get caught.
diff --git a/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc b/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
index 46793acff4..5cfad06af8 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
@@ -27,17 +27,21 @@ Integration tests for `tika-app` to be run from a distribution ZIP.
mkdir -p /tmp/tika-app-test
cd /tmp/tika-app-test
-# Copy and extract distribution
+# Copy and extract distribution. The zip has no wrapping directory: the jar,
+# lib/ and plugins/ land directly in the current directory.
cp /path/to/tika-app-4.0.0-SNAPSHOT.zip .
unzip tika-app-4.0.0-SNAPSHOT.zip
-cd tika-app-4.0.0-SNAPSHOT
# Get test files
-cp /path/to/tika-main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/test-documents/testPDF.pdf .
-cp /path/to/tika-main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/test-documents/test_recursive_embedded.docx .
-cp /path/to/tika-main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/test-documents/testHTML.html .
+cp /path/to/tika-main/release-tools/uat/test-files/testPDF.pdf .
+cp /path/to/tika-main/release-tools/uat/test-files/test_recursive_embedded.docx .
+cp /path/to/tika-main/release-tools/uat/test-files/testHTML.html .
----
+Below, `tika-app.jar` stands for the versioned jar the zip actually contains
+(`tika-app-4.0.0-SNAPSHOT.jar`). Run every command from the unzipped directory so the
+sibling `lib/` and `plugins/` are found.
+
== Test Cases
=== Test 1: Basic Text Extraction
@@ -147,7 +151,7 @@ java -jar tika-app.jar -J -t /tmp/tika-app-test/batch-input /tmp/tika-app-test/b
ls /tmp/tika-app-test/batch-output2
----
-*Expected:* Creates .json files with text content (tk:content-handler should be ToTextContentHandler).
+*Expected:* Creates .json files with text content (`tk:content-handler-type` should be `TEXT`).
=== Test 11: Version Check
@@ -244,7 +248,7 @@ Create `/tmp/tika-app-test/my-config.json`:
"basic-content-handler-factory": {
"type": "TEXT",
"writeLimit": 100000,
- "throwOnWriteLimit": false
+ "throwOnWriteLimitReached": false
}
},
"parsers": [
@@ -373,7 +377,7 @@ java -jar tika-app.jar -T 30000 /tmp/tika-app-test/batch-input /tmp/tika-app-tes
[source,bash]
----
mkdir -p /tmp/tika-app-test/heap-out
-java -jar tika-app.jar -X 2g /tmp/tika-app-test/batch-input /tmp/tika-app-test/heap-out
+java -jar tika-app.jar --Xmx=2g /tmp/tika-app-test/batch-input /tmp/tika-app-test/heap-out
----
*Expected:* Forked processes use 2GB heap.
@@ -390,9 +394,11 @@ MalformedURLException: no protocol: nonexistent_file.pdf
Should say "File not found".
-=== Issue 2: INFO Message on Every Command
+=== Issue 2: INFO Message on Every Default-Config Command
-Every command prints an INFO message to stderr about convenience features. Use `2>/dev/null` to suppress.
+Every command that loads the default single-file config prints an INFO message to stderr about
+the non-default features TikaCLI turns on. Passing `--config=<file>` skips it; otherwise use
+`2>/dev/null` to suppress.
=== Issue 3: Config Dump Options Not Implemented
diff --git a/docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc b/docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc
index f5594f616a..56b641d03a 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc
@@ -60,9 +60,9 @@ This produces two distribution zips:
contains the tika-eval-app jar and its `lib/`. Unzip and run from
there.
-[source,bash]
+[source,bash,subs=attributes+]
----
-mkdir -p ~/tika-runtime && cd ~/tika-runtime
+mkdir -p <workdir>/tika-runtime && cd <workdir>/tika-runtime
unzip -q tika-app/target/tika-app-{tika-version}.zip -d tika-app
unzip -q tika-eval/tika-eval-app/target/tika-eval-app-{tika-version}.zip -d tika-eval
----
@@ -74,7 +74,7 @@ the unzipped `lib/` siblings) will fail at runtime with
== Step 1: prepare the corpus
Put the input files somewhere on disk. For this example we'll use
-`/data/corpus/`. Tika's pipes iterator walks the tree recursively;
+`<corpus>/`. Tika's pipes iterator walks the tree recursively;
files at any depth are processed.
== Step 2: write the two extraction configs
@@ -86,7 +86,7 @@ from defaults — typically just the digester and (for this kind of
A/B test) the field under test.
The two configs should differ in exactly *one* thing — the field
-under test. Everything else (digester, parsers, worker count) should
+under test. Everything else (digester, parsers, fork count) should
match.
=== Config A (baseline)
@@ -142,22 +142,22 @@ not specified takes its default.
== Step 3: run extractions
-[source,bash]
+[source,bash,subs=attributes+]
----
-cd ~/tika-runtime/tika-app
+cd <workdir>/tika-runtime/tika-app
# Extract A — baseline
java -jar tika-app-{tika-version}.jar \
- -i /data/corpus \
- -o /data/extracts-A \
- -c /path/to/tika-config-A.json \
+ -i <corpus> \
+ -o <workdir>/extracts-A \
+ -c <workdir>/tika-config-A.json \
-n 8
# Extract B — variant
java -jar tika-app-{tika-version}.jar \
- -i /data/corpus \
- -o /data/extracts-B \
- -c /path/to/tika-config-B.json \
+ -i <corpus> \
+ -o <workdir>/extracts-B \
+ -c <workdir>/tika-config-B.json \
-n 8
----
@@ -166,28 +166,31 @@ Flags:
* `-i` / `--inputDir` — input directory (recursive walk)
* `-o` / `--outputDir` — output directory for per-file JSON
* `-c` / `--config` — optional config overrides
-* `-n` / `--numClients` — forked worker processes (default 4)
+* `-n` / `--numClients` — number of forked JVMs (default 2 for the `-i`/`-o`
+ form; 1 for the bare `<inputDir> <outputDir>` form)
* `-p` / `--pluginsDir` — explicit plugins directory (omit when
running from the unzipped distribution — auto-discovery handles it)
* `-T` / `--timeoutMillis` — total task timeout per file (sets
`totalTaskTimeoutMillis`)
-Each run forks worker processes and produces one `.json` per input
+Each run forks JVMs and produces one `.json` per input
file under the output directory. Run them sequentially or in
parallel — they're independent.
Progress is logged to stderr. On a typical workstation, extraction
throughput is in the hundreds-of-files-per-minute range, depending on
-parser cost and worker count.
+parser cost and fork count.
== Step 4: compare and report
-[source,bash]
+[source,bash,subs=attributes+]
----
-java -jar tika-eval/tika-eval-app/target/tika-eval-app-{tika-version}.jar \
+cd <workdir>/tika-runtime/tika-eval
+
+java -jar tika-eval-app-{tika-version}.jar \
-n 8 -r -d my-comparison \
- -a /data/extracts-A \
- -b /data/extracts-B
+ -a <workdir>/extracts-A \
+ -b <workdir>/extracts-B
----
The `Compare` subcommand keyword is optional — the CLI infers it from
@@ -202,12 +205,12 @@ Options:
* `-i` / `--inputDir` — original binary input directory (optional, lets
tika-eval pair extracts to source files even if A or B failed on some)
* `-d` / `--db` — H2 database name/path. A short label is fine —
- tika-eval will create `\{label}.mv.db` and a `\{label}-reports/` dir
- alongside. Persist the db if you want to re-run Report later.
+ tika-eval will create `\{label}.mv.db`. Persist the db if you want to
+ re-run Report later.
* `-r` / `--report` — automatically run the Report step after Compare,
and tgz the reports directory to `<reportsDir>.tgz`.
-* `-rd` / `--reportsDir` — explicit reports directory (overrides the
- default derived from `-d`).
+* `-rd` / `--reportsDir` — reports directory; defaults to `reports` in the
+ current directory, not to anything derived from `-d`.
* `-z` / `--gzip` — gzip the H2 db file to `<db>.mv.db.gz` after Compare
so it can be transferred. Requires `-d` (no-op with a warning for a
temp db or a non-file jdbc connection (e.g. `jdbc:h2:mem:`, tcp)). Combine with `-r` to package
@@ -223,13 +226,15 @@ The reports directory contains subdirectories:
** `mime_diffs_A_to_B.xlsx` — summary counts per (A type, B type)
transition
** `mime_diffs_A_to_B_details.xlsx` — one row per file that changed
+* `charset/` — detected-encoding coverage, distribution, and A→B flips
* `content/` — token-count and content comparison
* `exceptions/` — new/changed exceptions in B vs A
* `attachments/` — embedded-document comparison
* `metadata/` — metadata comparison
+* `md5/`, `parse_times/` — digest and timing comparisons
-Open the `.xlsx` files directly, or use the `regression` skill
-(`~/.claude/skills/regression/SKILL.md`) for guided triage.
+Open the `.xlsx` files directly, or query the H2 database for custom counts and
+joins (see <<h2>>).
== Tips
@@ -247,24 +252,25 @@ Open the `.xlsx` files directly, or use the `regression` skill
parsers may corrupt input files in rare cases; keep a known-good
source tree separate from working copies.
-* *Persist the H2 database (`-d /path/to/eval.db`)* if you want to
+* *Persist the H2 database (`-d <workdir>/eval.db`)* if you want to
re-run the Report step or query it directly with the H2 console.
Omitting `-d` writes to a temp file that's cleaned up on exit.
* *Re-run only Report*: if you already have the database, skip Compare
and run the Report tool directly:
+
-[source,bash]
+[source,bash,subs=attributes+]
----
-java -jar tika-eval/tika-eval-app/target/tika-eval-app-{tika-version}.jar Report \
- -d /tmp/eval.db -rd /data/reports/
+cd <workdir>/tika-runtime/tika-eval
+java -jar tika-eval-app-{tika-version}.jar Report \
+ -d <workdir>/eval.db -rd <workdir>/reports/
----
== Worked example: encoding-detector chain comparison
Compare the SPI default detector chain (A) against an explicit
`[BOM, HTML, Mojibuster, JunkFilter]` chain (B) over a 29 000-file
-Common Crawl HTML sample at `~/data/commoncrawl/cc-html-sample`.
+Common Crawl HTML sample at `<corpus>/cc-html-sample`.
. One-time build: `./mvnw install -pl tika-app,tika-eval/tika-eval-app -am -DskipTests`
. Unzip both distributions: `unzip tika-app/target/tika-app-*.zip` and
@@ -274,49 +280,52 @@ Common Crawl HTML sample at `~/data/commoncrawl/cc-html-sample`.
examples above.
. Extract A:
+
-[source,bash]
+[source,bash,subs=attributes+]
----
-cd ~/tika-runtime/tika-app
+cd <workdir>/tika-runtime/tika-app
java -jar tika-app-{tika-version}.jar \
- -i ~/data/commoncrawl/cc-html-sample \
- -o ~/data/extracts/cc-html-sample-A \
- -c ~/configs/tika-config-A.json -n 8
+ -i <corpus>/cc-html-sample \
+ -o <workdir>/extracts/cc-html-sample-A \
+ -c <workdir>/configs/tika-config-A.json -n 8
----
. Extract B (same form, different config + output):
+
-[source,bash]
+[source,bash,subs=attributes+]
----
java -jar tika-app-{tika-version}.jar \
- -i ~/data/commoncrawl/cc-html-sample \
- -o ~/data/extracts/cc-html-sample-B \
- -c ~/configs/tika-config-B.json -n 8
+ -i <corpus>/cc-html-sample \
+ -o <workdir>/extracts/cc-html-sample-B \
+ -c <workdir>/configs/tika-config-B.json -n 8
----
. Compare + report:
+
-[source,bash]
+[source,bash,subs=attributes+]
----
-cd ~/tika-runtime/tika-eval
+cd <workdir>/tika-runtime/tika-eval
java -jar tika-eval-app-{tika-version}.jar \
-n 8 -r -d cc-html-29k-A-vs-B \
- -a ~/data/extracts/cc-html-sample-A \
- -b ~/data/extracts/cc-html-sample-B
+ -a <workdir>/extracts/cc-html-sample-A \
+ -b <workdir>/extracts/cc-html-sample-B
----
+
-Produces `cc-html-29k-A-vs-B-reports/` plus a `.tgz` of the same
-alongside `cc-html-29k-A-vs-B.mv.db`.
+Produces `reports/` plus `reports.tgz` in the current directory, alongside
+`cc-html-29k-A-vs-B.mv.db`. Pass `-rd cc-html-29k-A-vs-B-reports` if you want
+the reports directory named after the run.
-For a 29 K-file HTML sample on a typical workstation (8 forked workers,
+For a 29 K-file HTML sample on a typical workstation (8 forks,
i9-ish CPU): each extraction completes in roughly 45-60 seconds; the
Compare step finishes in 2-5 minutes depending on extract size.
. Open `reports/mimes/mime_diffs_A_to_B.xlsx` to see the headline
- MIME-detection differences; the encoding-detector chain change
- surfaces as charset diffs in `mimes/mime_diffs_A_to_B_details.xlsx`
- (`tk:detected-encoding`).
+ MIME-detection differences. The encoding-detector chain change shows up
+ in `reports/charset/` — `charset_coverage.xlsx` counts the A→B flips, and
+ the distribution reports break down `tk:detected-encoding` per run. It also
+ shows up in `mimes/mime_diffs_A_to_B_details.xlsx`, because the recorded
+ MIME string carries the `charset=` parameter.
-A regression analysis writeup goes in
-`~/Desktop/claude-todo/<reports-dir-name>-analysis.md` per the
-`regression` skill.
+Keep the writeup next to the reports directory it describes, so the numbers and
+the narrative travel together.
+[[h2]]
== Querying the H2 database directly
`tika-eval` stores everything in an *H2* database — the `-d <name>` passed to
@@ -324,8 +333,8 @@ A regression analysis writeup goes in
views; for custom counts and joins, connect to the H2 file and run SQL.
The database is created with *no username and no password*
-(`H2Util` opens it with `DriverManager.getConnection(url)` and no credentials).
-So the old H2 default `-user sa` *fails* with "Wrong user name or password" — use
+(`H2Util` opens it with `DriverManager.getConnection(url)` and no credentials),
+so `-user sa` *fails* with "Wrong user name or password" — use
empty credentials. Append `;IFEXISTS=TRUE` (so a typo opens nothing rather than
creating a new empty db) and `;ACCESS_MODE_DATA=r` for a read-only open. H2 is
single-writer: query only after the `Compare`/`Profile`/`Report` run has released
diff --git a/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc b/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
index ec3db034eb..9842d34ec8 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
@@ -146,7 +146,9 @@ curl -s -X PUT -T test_recursive_embedded.docx http://localhost:9998/rmeta/text
curl -s -X PUT -T testPDF.pdf http://localhost:9998/language
----
-*Expected:* Two-letter language code (e.g., `en`, `th`).
+*Expected:* An ISO 639 code, i.e. 2-3 lowercase letters. Do not assert a *particular*
+language here: `testPDF.pdf` carries too little text for detection to be reliable, which
+is why `run-uat.sh` matches `^[a-z]{2,3}$` rather than a specific code.
=== Test 12: PUT /unpack/all
@@ -237,7 +239,7 @@ EOF
java -jar tika-server-standard-4.0.0-SNAPSHOT.jar -c tika-config-pipes-no-flags.json
----
-*Expected:* The server refuses to start, failing with a `TikaConfigException` stating that the `pipes` endpoint requires `allowPipes` to be `true`. The same applies to `async`. (`/status` is no longer gated — it can be enabled simply by listing `status` under `endpoints`.)
+*Expected:* The server refuses to start, failing with a `TikaConfigException` stating that the `pipes` endpoint requires `allowPipes` to be `true`. The same applies to `async`. `/status` is not gated — listing `status` under `endpoints` is enough.
== Part 2: Tests with allowPipes / allowPerRequestConfig
@@ -351,14 +353,20 @@ java -jar tika-server-standard-4.0.0-SNAPSHOT.jar -c tika-config.json &
== Headers
-=== Test 26: X-Tika-OCRskipOcr Header
+=== Test 26: Removed X-Tika-* headers are ignored
+
+The `X-Tika-OCR*` and `X-Tika-PDF*` header families no longer exist — `grep X-Tika`
+over `tika-server/` finds nothing. This test only confirms that sending one is inert
+rather than an error; it does *not* change OCR behaviour.
[source,bash]
----
curl -s -X PUT -H "X-Tika-OCRskipOcr: true" -T testPDF.pdf http://localhost:9998/tika/text
----
-*Expected:* Text extraction without OCR.
+*Expected:* HTTP 200 with the same body as the same request without the header. To
+configure OCR in 4.x, use `tika-config.json` (or per-request config where enabled) —
+see xref:migration-to-4x/migrating-tika-server-4x.adoc[Migrating tika-server to 4.x].
=== Test 27: Content-Disposition Filename
@@ -501,8 +509,8 @@ The server automatically configures the required fetcher and emitter for pipes-b
Both temp directories are cleaned up on server shutdown.
-If a user config file does not include `plugin-roots`, the server automatically adds a default value pointing to a `plugins` directory in the current working directory.
+If a user config file does not include `plugin-roots`, the server automatically adds a default value: a `plugins` directory beside the server jar if one exists, otherwise `plugins` in the current working directory. An explicit `plugin-roots` in the user config is never overwritten.
=== Security Boundary
-Child processes (pipes workers) are configured with `basePath` rather than `allowAbsolutePaths`, ensuring they can only access files within their designated temp directories. This provides a security boundary between the parent server process and forked child processes.
+Forks are configured with `basePath` rather than `allowAbsolutePaths`, ensuring they can only access files within their designated temp directories. This provides a security boundary between the parent server process and its forks.
diff --git a/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc b/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
index c5397a7464..e163696419 100644
--- a/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
+++ b/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
@@ -17,77 +17,110 @@
= Building the Junk Detector
-This page documents the training pipeline, model format, evaluation methodology,
-and guidance for improving the junk detector model. For usage, see
+The training pipeline, model format, and known limitations of the bundled
+junk-detector model. For usage, see
xref:advanced/junk-detection.adoc[Text Quality Scoring (Junk Detection)].
-== Overview
+Corpus paths below are written as `<corpus>` and `<workdir>`; substitute your own.
-The junk detector is a per-script byte-bigram language model. For each
-Unicode script (Latin, Cyrillic, Arabic, Han, etc.) it maintains a 256×256
-table of `log P(byte_b | byte_a)` values — the probability of seeing byte `b`
-immediately after byte `a` in clean UTF-8 text of that script.
+== Model shape
-The pipeline has three stages:
+The model scores text with nine features and one global linear combiner:
+
+[cols="1,4"]
+|===
+| Feature | What it measures
+
+| z1
+| Codepoint-bigram log-probability. Every bigram is bucketed to its Unicode
+ script and scored against that script's own table; unseen pairs fall back to a
+ unigram independence score. *The only per-script feature*, and the only one
+ with per-script mu/sigma calibration.
+
+| z2
+| Unicode block-transition log-probability. One global table.
+
+| z3
+| Control-byte fraction over [0x01–0x08, 0x0B, 0x0C, 0x0E–0x1F, 0x7F].
+
+| z4
+| Script-transition log-probability. One global table over raw
+ `Character.UnicodeScript` values.
+
+| z5–z9
+| Document-level quality: letter-adjacent-to-mark ratio, replacement-character
+ ratio, script density, script coherence, script alternation.
+|===
+
+`logit = w1*z1 + … + w9*z9 + bias`, fit on clean versus corrupted windows.
+Positive means clean; the natural threshold is 0. The bundled model covers 72
+scripts (including COMMON, which is treated as a script in its own right so that
+digits and punctuation do not pollute the neighbouring script's evidence).
+
+== Pipeline
[source]
----
-1. BuildJunkTrainingData — collect and split corpus per script group
-2. TrainJunkModel — train bigram tables and calibrate z-scores
-3. EvalJunkDetector — measure discrimination quality
+1. BuildJunkTrainingData — collect and split corpus per script group
+2. BuildJunkAugmentationData — optional: fold in real-document text
+3. TrainJunkModel — train tables, calibrate, write the model
----
-All three tools are packaged as a fat JAR via the `train` Maven profile:
+Tools live in `tika-ml/tika-ml-junkdetect-tools` and are packaged as a
+self-contained jar by the `train` profile:
[source,bash]
----
-mvn -pl tika-ml/tika-ml-junkdetect package -Ptrain -DskipTests
+./mvnw clean package -pl tika-ml/tika-ml-junkdetect-tools -am -Ptrain -DskipTests
----
-The resulting JAR is `tika-ml-junkdetect-*-train.jar`.
+The result is `tika-ml-junkdetect-tools-*-tools.jar`.
+
+[IMPORTANT]
+====
+Durable training parameters are *not* CLI flags. They are `public static final`
+constants in `JunkDetectorTrainingConfig`, and both tools **refuse to start** if a
+CLI argument tries to override one. A model file's identity therefore always
+matches a committed config: to change a parameter, edit that file and commit the
+change together with the retrained model.
+====
-== Stage 1: Corpus collection (`BuildJunkTrainingData`)
+== Stage 1: corpus collection (`BuildJunkTrainingData`)
-This tool collects clean UTF-8 sentences from language-specific source files,
-groups them by Unicode script, allocates a byte budget proportional to
-per-script bigram entropy, and writes 80/10/10 train/dev/test splits.
+Collects clean UTF-8 sentences from language-specific source files, groups them by
+Unicode script, allocates a byte budget per script, and writes 80/10/10
+train/dev/test splits.
=== Data format
-Source data lives in one directory per language (ISO 639 code), each containing
-up to two files:
+One directory per language (ISO 639 code), each containing up to two files:
`sentences_wikipedia.txt`::
- Line-numbered Wikipedia sentences: `\{lineNum}\{TAB}\{text}`.
- One sentence per line.
+ Line-numbered Wikipedia sentences: `\{lineNum}\{TAB}\{text}`, one per line.
`sentences_madlad.txt`::
- Line-numbered MADLAD-400 documents: `\{lineNum}\{TAB}\{text}`.
- Documents contain literal two-character `\n` escape sequences as
- sub-sentence separators. The tool splits on these before processing.
+ Line-numbered MADLAD-400 documents, same shape. Documents use literal
+ two-character `\n` escape sequences as sub-sentence separators; the tool splits
+ on those first.
=== Script group detection
-For each language directory the dominant Unicode script is detected by
-sampling up to 2,000 lines and histogramming `Character.UnicodeScript` over
-all codepoints. The `COMMON`, `INHERITED`, and `UNKNOWN` pseudo-scripts are
-excluded. The plurality script (with a 1% minimum floor to suppress spurious
-wins on mixed-script text) determines which group that language belongs to.
-
-Languages that share the same dominant script are pooled together into one
-training group. No script groups are hardcoded — the set of groups is derived
-entirely from the data.
+Each language directory's dominant script is found by sampling up to 2,000 lines
+and histogramming `Character.UnicodeScript`, excluding the `COMMON`, `INHERITED`,
+and `UNKNOWN` pseudo-scripts. The plurality script wins, subject to a 1% floor
+that suppresses spurious wins on mixed-script text. Languages sharing a dominant
+script are pooled into one training group. No groups are hardcoded — the set is
+derived from the data.
=== Entropy-proportional byte budget
-All scripts are not equal: CJK text has thousands of distinct 3-byte UTF-8
-codepoints producing high byte-bigram entropy (~10.4 bits), while Arabic text
-clusters in a narrow 0xD8–0xDB high-byte range (~7.2 bits). A naïve
-sentence-count budget would badly over-represent low-entropy scripts.
+Scripts are not comparable by sentence count: CJK text has thousands of distinct
+3-byte UTF-8 codepoints and high byte-bigram entropy (~10.4 bits), while Arabic
+clusters in a narrow 0xD8–0xDB high-byte range (~7.2 bits). A sentence-count
+budget would badly over-represent low-entropy scripts.
-Instead the tool allocates a **total byte budget** (default 50 MB) across
-script groups in proportion to their empirical byte-bigram Shannon entropy,
-estimated from a 200 KB sample per group:
+The tool instead splits the total byte budget in proportion to each group's
+empirical byte-bigram Shannon entropy, estimated from a 200 KB sample per group:
[source]
----
@@ -96,14 +129,12 @@ H(script) = -Σ p(a,b) · log₂ p(a,b) over all observed bigrams (a,b)
budget(script) = totalBudget × H(script) / Σ H(all scripts)
----
-Within each script group the budget is distributed evenly across its member
-languages, ensuring no single language dominates the training data.
+Within a group the budget is spread evenly across member languages, and a
+per-language cap keeps one large source (`zho` has 8 GB of MADLAD) from dominating
+a multi-language bucket.
=== Train/dev/test split
-After collecting and shuffling sentences, the tool writes three gzipped files
-per script:
-
[cols="1,1,3"]
|===
| File | Split | Purpose
@@ -114,315 +145,247 @@ per script:
| `\{script}.dev.gz`
| 10%
-| Calibration (mu/sigma estimation) in `TrainJunkModel`.
- Also used for iterative evaluation during development.
+| Calibration (mu/sigma). Also the split to use for iterative evaluation.
| `\{script}.test.gz`
| 10%
-| **Held out completely.** Use only for final reported evaluation numbers.
- Never use to make model or threshold decisions.
+| **Held out completely.** Final reported numbers only — never a model or
+ threshold decision.
|===
-=== Running corpus collection
-
-[source,bash]
-----
-java -cp tika-ml-junkdetect-*-train.jar \
- org.apache.tika.ml.junkdetect.tools.BuildJunkTrainingData \
- --data-dir ~/datasets/madlad/data \
- --output-dir ~/datasets/madlad/junkdetect \
- --total-budget-bytes 50000000
-----
-
-Key options:
-
-[cols="2,1,3"]
-|===
-| Option | Default | Description
-
-| `--data-dir`
-| `~/datasets/madlad/data`
-| Root directory containing per-language subdirectories.
-
-| `--output-dir`
-| `~/datasets/madlad/junkdetect`
-| Where to write `\{script}.train.gz`, `.dev.gz`, `.test.gz`, and `manifest.tsv`.
-
-| `--total-budget-bytes`
-| `50000000`
-| Total UTF-8 byte budget across all scripts. Increase for production runs.
-
-| `--min-bytes`
-| `50`
-| Minimum UTF-8 byte length for a sentence to be accepted.
-
-| `--max-punc-frac`
-| `0.30`
-| Maximum fraction of codepoints that may be ASCII punctuation or digits.
- Filters out bullet lists, code snippets, and other non-prose content.
-
-| `--seed`
-| `42`
-| Random seed for reproducible shuffles.
-
-| `--dry-run`
-| `false`
-| Print script detection and entropy results without writing files.
-|===
-
-== Stage 2: Training (`TrainJunkModel`)
-
-For each script, this tool reads the `.train.gz` file, accumulates
-byte-bigram counts, applies Laplace smoothing, computes log-probabilities,
-then calibrates z-score statistics from the `.dev.gz` file.
-
-=== Bigram table training
-
-[source,subs="-attributes"]
-----
-for each sentence in {script}.train.gz:
- utf8 = sentence.getBytes(UTF-8)
- for each consecutive pair (a, b) in utf8:
- counts[a * 256 + b]++
-
-for each row a in 0..255:
- rowTotal = Σ (counts[a * 256 + b] + 1) for b in 0..255 // Laplace add-1
- for each b in 0..255:
- table[a * 256 + b] = log((counts[a * 256 + b] + 1) / rowTotal)
-----
-
-Laplace (add-1) smoothing is applied per row: every possible next byte is
-given a pseudocount of 1, preventing log(0) for unseen bigrams and providing
-a small but nonzero probability for novel byte sequences.
-
-=== Calibration
-
-For each sentence in `\{script}.dev.gz`:
-
-[source]
-----
-meanLogProb = Σ table[bigram] / (bytes - 1)
-----
-
-The calibration statistics are the mean (μ) and standard deviation (σ) of
-`meanLogProb` across all dev sentences. At inference:
+`manifest.tsv` records per-script entropy, budget, bytes written, sentence count,
+and contributing languages.
-[source]
-----
-zScore = (meanLogProb - μ) / σ
-----
-
-A z-score of 0 means "exactly as likely as average clean text for this script."
-Negative scores indicate text that is less likely than clean — i.e., garbled.
-
-=== Running training
+=== Running it
[source,bash]
----
-java -cp tika-ml-junkdetect-*-train.jar \
- org.apache.tika.ml.junkdetect.tools.TrainJunkModel \
- --data-dir ~/datasets/madlad/junkdetect \
- --output ~/datasets/madlad/junkdetect/junkdetect.bin
-----
-
-After training, copy the model to the classpath resource location:
-
-[source,bash]
-----
-cp ~/datasets/madlad/junkdetect/junkdetect.bin \
- tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
+java -cp tika-ml-junkdetect-tools-*-tools.jar \
+ org.apache.tika.ml.junkdetect.tools.BuildJunkTrainingData \
+ --data-dir <corpus>/madlad/data \
+ --output-dir <workdir>/junkdetect
----
-== Stage 3: Evaluation (`EvalJunkDetector`)
+`--data-dir`, `--output-dir`, and `--dry-run` (detect scripts and show the budget
+without writing) are the only accepted arguments.
-The evaluator measures how well the model separates clean text from
-corrupted text across scripts, distortion types, and string lengths.
+=== Config constants
-=== Distortion modes
+`JunkDetectorTrainingConfig` holds everything else:
-[cols="1,3"]
+[cols="3,1,4"]
|===
-| Mode | Description
-
-| `inject`
-| Random bytes (0x80–0xFF) are substituted at rate `r` of positions.
- Tests from 1% injection (subtle corruption) to 90% (nearly all garbage).
-
-| `char-reverse`
-| Codepoints are reversed (Unicode-aware, preserving surrogate pairs).
- Produces valid UTF-8 but in nonsensical reading order.
- Most meaningful for RTL scripts (Arabic, Hebrew) where reversed text
- is a realistic failure mode; LTR script bigrams are nearly symmetric,
- so detection is harder.
-
-| `byte-shuffle`
-| All bytes are randomly shuffled (Fisher-Yates).
- The most extreme corruption — destroys all sequential structure.
+| Constant | Value | Effect
+
+| `TOTAL_BUDGET_BYTES`
+| 500,000,000
+| Total UTF-8 byte budget across all script groups.
+
+| `PER_LANGUAGE_CAP_BYTES`
+| 5,000,000
+| Cap on one language's contribution to a multi-language bucket. Single-language
+ buckets ignore it.
+
+| `MIN_TARGET_SCRIPT_FRAC`
+| 0.05
+| Minimum fraction of non-COMMON codepoints that must be in the bucket's target
+ script. Low enough to keep legitimate mixed-script content (kanji + kana,
+ Korean with hanja), high enough to reject off-target lines.
+
+| `MIN_BYTES_PER_SENTENCE`
+| 50
+| Minimum UTF-8 length for a sentence to pass the quality filter.
+
+| `MAX_PUNC_FRAC`
+| 0.30
+| Maximum fraction of ASCII punctuation and digits. Filters bullet lists and
+ code snippets.
+
+| `MIN_DEV_SENTENCES`
+| 500
+| A script below this in the dev split is excluded — too little data for reliable
+ calibration, which inflates FPR. At `DEV_FRAC = 0.10` that is ~5,000 sentences
+ total.
+
+| `SCRIPT_SAMPLE_LINES`
+| 2,000
+| Lines sampled per language for dominant-script detection.
+
+| `ENTROPY_SAMPLE_BYTES`
+| 200,000
+| Bytes sampled per group for the entropy estimate.
+
+| `SEED`
+| 42
+| Shuffle seed.
+
+| `DROP_SCRIPTS`
+| GOTHIC, THAANA
+| Script buckets to exclude from the next build: THAANA has 216 native train
+ sentences, and 40% of the Wikipedia "gothic" directory is English text *about*
+ Gothic. The javadoc also expects `MIN_TARGET_SCRIPT_FRAC` to remove
+ CANADIAN_ABORIGINAL, CHEROKEE, and TIFINAGH. None of that has been applied yet:
+ the bundled `junkdetect.bin` predates these settings and still carries tables for
+ all five, so `knownScripts()` returns them today.
+
+| `MIN_BIGRAM_COUNT`
+| 3
+| Drop z1 bigrams occurring fewer than 3 times in a script. Singletons and
+ doubletons are overwhelmingly OCR artifacts and proper-noun noise that inflate
+ the clean-side tail without adding signal. Set to 1 to disable.
+
+| `OA_LOAD_FACTOR`
+| 0.5
+| Target load factor for the per-script open-addressed table: ~2 probes per
+ lookup.
+
+| `KEY_INDEX_BITS`
+| 16
+| Bits per codepoint index in a packed bigram key. Supports 65,535 codepoints per
+ script; HAN is the worst case at ~15 K.
|===
-=== Output files
-
-`detail.tsv`::
- One row per `(script, distortion, param, length)` cell, with columns:
- `script`, `distortion`, `param`, `length`, `n_clean`, `n_corrupt`,
- `mean_clean_z`, `mean_corrupt_z`, `cohens_d`, `fpr`, `tpr`.
-
-`summary.tsv`::
- Macro-averaged across scripts per `(distortion, param, length)`.
- The `macro_cohens_d` column is the headline comparison metric.
+== Stage 2: augmentation (`BuildJunkAugmentationData`, optional)
-=== Key metrics
-
-**Cohen's d** (primary metric)::
- Effect size separating clean from corrupted z-scores:
-+
-[source]
-----
-d = (mean_clean_z - mean_corrupt_z) / pooled_std
-----
-+
-Higher is better. A value of 1.0 means the distributions are separated by
-one pooled standard deviation. Values above 2.0 indicate strong, reliable
-discrimination.
+The Wikipedia/MADLAD corpus is clean linguistic text and carries almost no HTML
+symbols (©, ®, ™, €, £). On real web pages those bytes then look anomalously
+surprising, which tips charset arbitration toward whichever encoding happens to
+put a frequent training letter at the same byte position.
-**True positive rate (TPR)**::
- Fraction of corrupted samples with z < threshold (−2.0 by default).
- Higher is better.
+`BuildJunkAugmentationData` folds quality-filtered text from tika-app RMETA JSON
+output into the `.train.gz` files to close that gap. It is strictly additive: the
+baseline directory is never modified, `.dev.gz` / `.test.gz` are copied verbatim
+so evaluation stays honest, and the tool refuses to run if the output directory
+resolves to the baseline. Per-script gates require a minimum document count and
+cap appended lines as a fraction of the baseline, so augmentation cannot skew the
+distribution toward data-rich scripts.
-**False positive rate (FPR)**::
- Fraction of clean samples with z < threshold. Should stay near 2–5%.
- A well-calibrated model will have FPR ≈ 2.5% (since z < −2.0 corresponds
- to the left tail of the standard normal for clean text).
+== Stage 3: training (`TrainJunkModel`)
-=== Running evaluation
+Reads each script's `.train.gz`, accumulates codepoint-bigram counts, builds the
+quantized per-script tables, then calibrates z1's mu and sigma from `.dev.gz`.
+The global tables (z2, z4) and the global calibrations (z3, z5–z9) and combiner
+weights are fit in the same run.
[source,bash]
----
-# During development: use the dev split
-java -cp tika-ml-junkdetect-*-train.jar \
- org.apache.tika.ml.junkdetect.tools.EvalJunkDetector \
- --data-dir ~/datasets/madlad/junkdetect \
- --split dev \
- --output-dir ~/datasets/madlad/junkdetect/eval
-
-# Final reporting only: use the held-out test split
-java -cp tika-ml-junkdetect-*-train.jar \
- org.apache.tika.ml.junkdetect.tools.EvalJunkDetector \
- --data-dir ~/datasets/madlad/junkdetect \
- --split test \
- --output-dir ~/datasets/madlad/junkdetect/eval-final
+java -cp tika-ml-junkdetect-tools-*-tools.jar \
+ org.apache.tika.ml.junkdetect.tools.TrainJunkModel
----
-IMPORTANT: Use `--split test` only once, for final reporting. The test split
-is completely held out and should never inform model or threshold decisions.
+The tool takes **no arguments**. Its input directory and its output path — the
+classpath resource
+`tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin`
+— are baked into `main`, deliberately: one bundled model in git, never a training
+run to a scratch directory, never parallel A/B variants. Editing either means
+editing and committing the source.
-=== Tracking improvement
-
-To compare two model versions:
-
-1. Train model A, run `EvalJunkDetector --split dev`, save `summary.tsv` as
- `summary-A.tsv`.
-2. Retrain as model B, run eval again, save as `summary-B.tsv`.
-3. Diff the `macro_cohens_d` column. Positive change = improvement.
+=== Calibration
-The `# OVERALL` line at the bottom of `summary.tsv` gives a single-number
-summary of model quality.
+For each dev sentence, `meanLogProb = Σ table[bigram] / (bigrams - 1)`; mu and
+sigma are its mean and standard deviation across the split. At inference
+`z = (meanLogProb - mu) / sigma`, so 0 means "exactly as likely as average clean
+text for this script" and negative means less likely than clean.
== Model binary format (JUNKDET1)
-The model is stored as a gzipped binary file. Auto-detection of the gzip
-wrapper is done by inspecting the first two bytes (magic `0x1f 0x8b`).
+Gzipped; the loader detects the wrapper from the first two bytes (`0x1f 0x8b`).
+The version byte is a hard gate — older files are rejected rather than read
+through a compatibility path, because parallel scoring paths are a known source of
+silent miscalibration.
[source]
----
[8 bytes] magic "JUNKDET1" (ASCII)
-[1 byte] version = 1
-[4 bytes] num_scripts (int32 big-endian)
-
-For each script (sorted by name):
- [2 bytes] name length (uint16 big-endian)
- [N bytes] script name (UTF-8)
- [4 bytes] μ — mean of dev-set mean_bigram_logprob (float32 big-endian)
- [4 bytes] σ — std deviation (float32 big-endian)
- [65536×4 bytes] log-prob table (float32 big-endian, index = a*256+b)
+[1 byte] version (15)
+[4 bytes] num_scripts (int32 BE)
+[1 byte] block_scheme_version
+
+// z4 — global script-transition section
+[1 byte] num_script_buckets
+ per bucket: [2 bytes] name length, [N bytes] name (UTF-8)
+[4 bytes] scriptTrans quant min (float32 BE)
+[4 bytes] scriptTrans quant max
+[num_script_buckets² × 2 bytes] transition table (int16 quantized)
+[8 bytes] z4 calibration {mu, sigma}
+
+// z2 — global block-transition section
+[4 bytes] block quant min
+[4 bytes] block quant max
+[block_N² × 2 bytes] transition table (int16 quantized)
+[8 bytes] z2 calibration {mu, sigma}
+
+// remaining global calibrations, {mu, sigma} float32 pairs
+[8 bytes] z3 (control-byte ratio)
+[8 bytes] z5 (letter-adjacent-to-mark)
+[8 bytes] z6 (replacement-char ratio)
+[8 bytes] z9 (script-alternation)
+
+// global combiner
+[1 byte] num_features
+[(num_features + 1) × 4 bytes] weights w1..wN then bias
+
+// per-script section, sorted by script name
+ [2 bytes] name length, [N bytes] name (UTF-8)
+ [8 bytes] z1 calibration {mu, sigma}
+ [variable] bigram + unigram tables: codepoint index, then sorted-occupied
+ bigram keys (first as int32 BE, rest as LEB128 varint deltas) with
+ 8-bit quantized bigram and unigram log-prob values
----
-The default classpath resource is
-`org/apache/tika/ml/junkdetect/junkdetect.bin`.
-
-== Known limitations and improvement paths
-
-=== Baltic and closely related Latin scripts
-
-The LATIN script pools ~322 languages from Latin, Basic Latin, and extended
-Latin alphabets. Baltic languages (Lithuanian, Latvian) use distinctive
-diacritics encoded differently in cp1257 vs. cp1252, but these bigrams are
-diluted by the large shared Latin vocabulary. The model correctly identifies
-the winner but with low delta (< 0.5), below the production confidence
-threshold of 1.0.
-
-**Possible improvements:**
-
-* Retrain with Baltic languages weighted more heavily within the LATIN group.
-* Split LATIN into LATIN-WEST and LATIN-EAST sub-models, where LATIN-EAST
- receives its own dedicated bigram table trained primarily on Baltic, Slavic
- Latin (Polish, Czech, Slovak), and Romanian.
+Default classpath resource: `org/apache/tika/ml/junkdetect/junkdetect.bin`.
-=== RTL script reversal
+== Known limitations
-For Arabic and Hebrew, codepoint-reversal is a realistic failure mode (text
-stored in the wrong visual order). The model detects this with moderate
-Cohen's d at lengths ≥ 50 characters. Shorter strings (15–30 characters)
-show weaker separation because there are too few bigrams to be statistically
-reliable.
+=== Baltic and closely related Latin languages
-**Possible improvement:** train a secondary short-text specialist model for
-RTL scripts using finer-grained features (trigrams or unigram frequency
-distributions).
+The LATIN bucket pools hundreds of languages. Baltic languages (Lithuanian,
+Latvian) use distinctive diacritics encoded differently in cp1257 and cp1252, but
+those bigrams are diluted by the shared Latin vocabulary. The model picks the
+right winner with a delta below the production confidence threshold of 1.0.
-=== Scaling up
+Candidate improvements: weight Baltic languages more heavily within the LATIN
+bucket, or split LATIN into a dedicated LATIN-EAST bucket trained primarily on
+Baltic, Slavic-Latin (Polish, Czech, Slovak), and Romanian.
-The default 50 MB byte budget is a proof-of-concept setting. For production:
+=== RTL script reversal on short text
-* Increase `--total-budget-bytes` to 500 MB or more.
-* Larger budgets improve calibration quality (tighter σ, more accurate μ)
- and reduce variance on infrequent bigrams.
-* The model binary grows only slightly (the 256×256 table is the same size
- regardless of training set size) — only calibration quality improves.
+For Arabic and Hebrew, codepoint reversal is a realistic failure mode (text stored
+in the wrong visual order). Separation is good at 50+ characters and weak at
+15–30, where there are too few bigrams to be stable. A short-text RTL specialist
+using finer-grained features (trigrams, or unigram frequency distributions) is the
+obvious next step.
== Smoke tests
-Five smoke tests in `JunkDetectorSmokeTest` verify the bundled model.
-All tests use the `TextQualityDetector` interface and return `TextQualityScore`
-or `TextQualityComparison` from `tika-core`.
+`JunkDetectorSmokeTest` verifies the bundled model through the `TextQualityDetector`
+interface, using `TextQualityScore` and `TextQualityComparison` from `tika-core`.
[cols="1,3"]
|===
| Test | What it checks
| `cleanVsGarbage`
-| Clean English `TextQualityScore` z-score > random high-byte garbage z-score.
- Garbage is decoded from ISO-8859-1 to produce a scoreable string.
+| Clean English scores above random high-byte garbage (decoded from ISO-8859-1 so
+ it is scoreable at all).
| `forwardVsReversedArabic`
-| Forward Arabic z-score > codepoint-reversed Arabic z-score.
- Reversal is done at codepoint (not byte) granularity, preserving valid Unicode.
+| Forward Arabic scores above codepoint-reversed Arabic. Reversal is at codepoint
+ granularity, so the text stays valid Unicode.
| `cp1252VsCp1257OnBalticText`
-| `compare()` returns `TextQualityComparison` picking cp1257 for Lithuanian text.
- Delta > 0.1 (weak; Baltic limitation documented above).
+| `compare()` picks cp1257 for Lithuanian text. The asserted floor is only
+ delta > 0.05 — weak, per the Baltic limitation above.
| `cp1252VsCp1251OnRussianText`
-| `compare()` picks cp1251 for Russian text. Delta > 1.0 (strong; Cyrillic
- bigrams are highly distinctive).
+| `compare()` picks cp1251 for Russian text. Delta > 1.0; Cyrillic bigrams are
+ highly distinctive.
| `cleanVsShuffledCjk`
-| Clean Japanese z-score > byte-shuffled Japanese z-score.
- Shuffled bytes are decoded as ISO-8859-1 to produce a scoreable string.
+| Clean Japanese scores above byte-shuffled Japanese.
|===
-NOTE: Codepoint reversal of LTR scripts (Russian, Latin) is **not** a useful
-smoke test — LTR byte-bigram distributions are nearly symmetric, so the model
-cannot reliably distinguish forward from reversed text. The Russian test uses
-codec comparison (cp1251 vs. cp1252) instead, which is the actual real-world
-failure mode for Cyrillic text.
+NOTE: Codepoint reversal is *not* a useful test for LTR scripts. Their byte-bigram
+distributions are nearly symmetric, so forward and reversed are barely
+distinguishable. The Russian test therefore uses codec comparison
+(cp1251 vs cp1252), which is the real-world failure mode for Cyrillic.
diff --git a/docs/modules/ROOT/pages/advanced/junk-detection.adoc b/docs/modules/ROOT/pages/advanced/junk-detection.adoc
index 5425b7a764..a39d1232ba 100644
--- a/docs/modules/ROOT/pages/advanced/junk-detection.adoc
+++ b/docs/modules/ROOT/pages/advanced/junk-detection.adoc
@@ -38,21 +38,24 @@ for that.
== How it works
-The scorer uses a per-script byte-bigram log-probability model trained on clean
-Wikipedia and MADLAD-400 text. For each input it:
-
-1. **Identifies the dominant Unicode script** (Latin, Cyrillic, Arabic, Han, etc.)
- by histogramming `Character.UnicodeScript` over all codepoints.
-2. **Looks up the script's bigram table** — a 256×256 matrix of
- `log P(byte_b | byte_a)` values trained on clean text for that script.
-3. **Computes a mean log-probability** across all consecutive byte pairs in the
- UTF-8 encoding of the input.
-4. **Z-scores the result** against calibration statistics (mean and standard
- deviation measured on a held-out set of clean text for the same script).
-
-The z-score is the primary output: a score of 0 means "exactly as expected for
-clean text of this script"; a score of −3 means "three standard deviations worse
-than clean"; a score of −10 means "almost certainly garbled."
+The model is trained on clean Wikipedia and MADLAD-400 text. For each input it:
+
+1. **Buckets every codepoint bigram to its Unicode script** (Latin, Cyrillic,
+ Arabic, Han, …), folding COMMON/INHERITED glue into the adjacent script, and
+ scores each against that script's own codepoint-bigram table. Per-script
+ scores are combined count-weighted, so mixed-script text is scored fairly
+ rather than by picking one script.
+2. **Adds document-level features**: Unicode block transitions, control-byte
+ fraction, script transitions, letter-adjacent-to-mark ratio,
+ replacement-character ratio, script density, coherence, and alternation.
+3. **Combines them with one global linear combiner** into a single score.
+
+The combined score is the primary output, and it is a *logit*, not a
+per-script z-score: positive means clean, negative means junk, and the natural
+threshold is 0. `getPClean()` is `sigmoid(score)`.
+
+See xref:advanced/junk-detection-build.adoc[Building the Junk Detector] for the
+feature list, calibration, and model format.
== Using the API
@@ -62,12 +65,12 @@ the Java `ServiceLoader` mechanism.
Add the dependency to your project:
-[source,xml]
+[source,xml,subs=attributes+]
----
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-ml-junkdetect</artifactId>
- <version>${tika.version}</version>
+ <version>{tika-version}</version>
</dependency>
----
@@ -92,46 +95,41 @@ at application startup.
[source,java]
----
TextQualityScore score = detector.score("The quick brown fox jumps over the lazy dog.");
-System.out.println(score.getZScore()); // e.g. -0.74 — within normal range
-System.out.println(score.getPClean()); // e.g. 0.32 — P(clean) via sigmoid
+System.out.println(score.getZScore()); // combiner logit; > 0 means clean
+System.out.println(score.getPClean()); // sigmoid of the same value
----
=== Interpreting the score
[cols="1,3"]
|===
-| Z-score range | Interpretation
+| Score | Interpretation
| > 0
-| Better than average clean text — high-quality, well-formed natural language.
+| Clean. The natural decision boundary.
-| −1 to 0
-| Within normal range for clean text. Most real documents fall here.
-
-| −1 to −2
-| Mildly degraded. May indicate noisy OCR, code-heavy text, or unusual domain
- language. Not necessarily junk.
+| 0 to −2
+| Leaning junk but not decisively. Noisy OCR, code-heavy text, or unusual domain
+ language lands here.
| < −2
-| Two or more standard deviations below clean. Worth investigating.
- A reasonable threshold for triggering re-OCR or re-encoding.
+| A conservative junk threshold. Reasonable trigger for re-OCR or re-decoding.
| < −5
-| Almost certainly garbled. Wrong charset decoding, byte-reversed content,
- or heavy corruption.
+| Almost certainly garbled: wrong charset, reversed content, or heavy corruption.
|===
The `TextQualityScore` also carries:
-* `getPClean()` — `sigmoid(z)`, a rough probability estimate in [0, 1] that the
- text is clean. Useful for ranking candidates; the absolute value is not
- calibrated as a true probability.
-* `getCiLow()` / `getCiHigh()` — 95% confidence interval on the z-score. Narrow
- on long texts, wide on short ones. Use these when making threshold decisions on
- short strings.
-* `getDominantScript()` — the Unicode script name used for scoring (e.g. `"LATIN"`,
- `"CYRILLIC"`, `"ARABIC"`, `"HAN"`). If `isUnknown()` is true, the dominant
- script had no model and scoring was not possible.
+* `getPClean()` — `sigmoid(score)`. The combiner is a fitted logistic, so this is
+ usable for ranking; do not read it as a precisely calibrated probability.
+* `getCiLow()` / `getCiHigh()` — the score plus/minus
+ `1.96 × sigma / sqrt(bigrams)` for the dominant script. Narrow on long texts,
+ wide on short ones; use them for threshold decisions on short strings.
+* `getDominantScript()` — the Unicode script name that carried the scoring (e.g.
+ `"LATIN"`, `"CYRILLIC"`, `"HAN"`), or `"NONE"` when no scoreable letter was
+ found. When `isUnknown()` is true, no script in the input had a model and no
+ score is available.
=== Comparing two candidates
@@ -152,16 +150,19 @@ String ascp1251 = new String(rawBytes, Charset.forName("cp1251"));
TextQualityComparison result = detector.compare("cp1252", ascp1252, "cp1251", ascp1251);
-System.out.println(result.winner()); // "A" or "B"
-System.out.println(result.delta()); // z-score separation between the two
+System.out.println(result.winner()); // the winning label: "cp1252" or "cp1251"
+System.out.println(result.delta()); // score separation between the two
-if (result.winner().equals("B") && result.delta() > 1.0) {
+if ("cp1251".equals(result.winner()) && result.delta() > 1.0) {
// cp1251 is confidently the better decoding
}
----
-The `delta()` is the absolute difference in z-scores between the two candidates.
-As a rough guide:
+An `isUnknown()` candidate is treated as neutral (0), not `-∞`: a decoding whose
+script the model does not know must not lose to a garbled-but-recognisable one.
+
+`delta()` is the absolute score difference between the two candidates. As a rough
+guide:
[cols="1,3"]
|===
@@ -183,50 +184,41 @@ As a rough guide:
=== Listing known scripts
-[source,java]
-----
-detector.knownScripts(); // returns Set<String>
-// e.g. [ARABIC, ARMENIAN, BENGALI, CYRILLIC, DEVANAGARI, GEORGIAN,
-// GREEK, GUJARATI, GURMUKHI, HAN, HANGUL, HEBREW, HIRAGANA,
-// KANNADA, KHMER, LAO, LATIN, MALAYALAM, MYANMAR, ORIYA,
-// SINHALA, TAMIL, TELUGU, THAANA, THAI, TIBETAN, ...]
-----
-
-If the dominant script of an input is not in this set, `score()` returns a
-`TextQualityScore` where `isUnknown()` is true and no z-score is available.
+`JunkDetector.knownScripts()` returns the `Set<String>` of scripts with a trained
+table — 72 in the bundled model, covering the major living scripts plus a long
+tail of historic ones. It is on `JunkDetector`, not on the `TextQualityDetector`
+interface, so reach for it only when you have the concrete class. If none of an input's scripts is in that set, `score()`
+returns a `TextQualityScore` with `isUnknown()` true and no usable score.
== Thresholds and operating points
-There is no universally correct threshold. The right cutoff depends on your
-content and tolerance for false positives (flagging good text as junk).
-
-**Starting points:**
+There is no universally correct threshold; it depends on your content and your
+tolerance for flagging good text as junk. Starting points:
-* **Trigger re-OCR**: z < −2.0 (catches ~95% of severe corruption while flagging
- ~2–5% of legitimate text on average, more for short strings).
-* **Charset tiebreaking**: prefer the candidate with the higher z-score when
- `delta() > 1.0`; abstain if `delta() < 0.5`.
-* **Training data filtering**: z < −1.5 to remove mojibake and bot-generated
- noise from NLP corpora.
+* **Trigger re-OCR**: score < −2.0.
+* **Charset tiebreaking**: take the higher-scoring candidate when
+ `delta() > 1.0`; abstain below 0.5.
+* **Training-data filtering**: score < −1.5 to strip mojibake and bot noise from
+ an NLP corpus.
-For short text (under ~50 UTF-8 bytes), use `getCiLow()` rather than `getZScore()`
-for threshold decisions, since the confidence interval widens substantially.
+Under ~50 UTF-8 bytes, threshold on `getCiLow()` rather than `getZScore()` — the
+interval widens substantially there.
== Limitations
-* **Script coverage**: only scripts with a trained model can be scored. Unknown
- scripts return `isUnknown() = true`.
-* **Short text**: scoring is unreliable below ~15 UTF-8 bytes. The model needs
- at least a few bigrams to produce a stable estimate.
-* **Closely related charsets in the same script pool**: the LATIN model is trained
- across hundreds of languages, which dilutes the signal for closely related
- Western European and Baltic encodings (e.g., cp1252 vs. cp1257 on Lithuanian
- text). The winner is usually correct, but delta may be small (< 0.5).
-* **Deliberately obfuscated text**: content designed to look like natural language
- (e.g. by adversarial padding) is not detected.
+* **Script coverage** — only scripts with a trained table can be scored; others
+ return `isUnknown()`.
+* **Short text** — unreliable below ~15 UTF-8 bytes; there are too few bigrams for
+ a stable estimate.
+* **Closely related charsets within one script** — the LATIN table pools hundreds
+ of languages, which dilutes the signal between neighbours like cp1252 and cp1257
+ on Lithuanian text. The winner is usually right, but `delta()` may be < 0.5.
+* **Deliberately obfuscated text** — content padded to look like natural language
+ is not detected.
== Further reading
-For training methodology, model format, evaluation harness, and guidance on
-improving the model, see
-xref:advanced/junk-detection-build.adoc[Building the Junk Detector].
+* xref:advanced/junk-detection-build.adoc[Building the Junk Detector] — training,
+ model format, limitations
+* xref:advanced/charset-detection-design.adoc[Charset Detection Pipeline] — where
+ `compare()` is used in production
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/flores-AUTOMATIC.log b/docs/modules/ROOT/pages/advanced/lang-detection/flores-AUTOMATIC.log
deleted file mode 100644
index c85c6e09bb..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/flores-AUTOMATIC.log
+++ /dev/null
@@ -1,536 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-=== Language Detection Comparison Report ===
-
-Test sentences: 203,381
-CharSoup ∩ OpenNLP: 105 languages, 104,684 sentences
-CharSoup ∩ Lingua: 71 languages, 70,784 sentences
-CharSoup ∩ Optimaize: 63 languages, 61,811 sentences
-
-Model heap (approx):
- CharSoup: ~0.0 MB
- OpenNLP: ~75.3 MB
- Lingua: ~0.1 MB (low accuracy mode)
- Optimaize: ~94.6 MB
-
-Coverage-adjusted accuracy — each detector scored on its own supported languages only
- (test sentences whose true language is not in a detector's covered set are skipped)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─ CS(ms) ON(ms) Li(ms) Opt(ms) CS sent/s
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------------------------------------------------------
-@20 65.04% 68.68% 74.87% 72.85% 76.35% 75.88% 84.87% 84.38% 1,869 1,330 19,333 558 68,811
-@50 73.72% 79.14% 86.09% 85.43% 90.99% 90.63% 94.44% 94.29% 2,300 1,671 25,074 3,125 55,917
-@100 75.32% 80.77% 90.25% 90.12% 95.43% 95.33% 96.51% 96.50% 2,208 2,247 40,560 3,194 58,246
-@150 75.53% 80.93% 90.98% 90.97% 96.15% 96.11% 96.72% 96.76% 2,292 3,649 61,505 3,665 56,112
-@200 79.21% 82.78% 91.11% 91.11% 96.23% 96.20% 96.75% 96.79% 3,166 4,531 58,536 4,726 40,622
-@500 79.22% 82.78% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 4,271 6,729 65,820 4,381 30,112
-full 79.22% 82.78% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 2,446 3,429 52,124 3,431 52,579
-
-Breadth-weighted accuracy — all 203 FLORES languages, unsupported languages score 0
- (penalises limited coverage; use this to compare total useful output across all inputs)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------
-@20 41.01% 43.43% 42.05% 40.71% 27.46% 27.53% 26.76% 26.89%
-@50 46.49% 50.04% 48.35% 47.74% 32.72% 32.88% 29.77% 30.04%
-@100 47.49% 51.07% 50.68% 50.36% 34.32% 34.58% 30.43% 30.75%
-@150 47.63% 51.18% 51.09% 50.84% 34.58% 34.86% 30.49% 30.83%
-@200 49.95% 52.35% 51.16% 50.91% 34.61% 34.90% 30.50% 30.84%
-@500 49.95% 52.35% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-full 49.95% 52.35% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-
-Strict accuracy — CharSoup ∩ OpenNLP (105 languages, 104,684 sentences)
- ── CharSoup ── ── OpenNLP ── CS(ms) OpenNLP(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 70.63% 72.31% 76.69% 74.07% 746 415 140,327
-@50 79.72% 83.52% 87.27% 86.37% 974 579 107,478
-@100 81.15% 85.19% 91.05% 90.79% 1,055 1,169 99,227
-@150 81.30% 85.36% 91.68% 91.55% 1,311 1,659 79,850
-@200 84.18% 86.81% 91.77% 91.65% 1,749 2,567 59,854
-@500 84.19% 86.81% 91.78% 91.67% 1,537 2,078 68,109
-full 84.19% 86.81% 91.78% 91.67% 1,201 1,823 87,164
-
-Strict accuracy — CharSoup ∩ Lingua (71 languages, 70,784 sentences)
- ── CharSoup ── ── Lingua ── CS(ms) Lingua(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 80.90% 79.54% 77.78% 76.80% 477 3,785 148,394
-@50 91.86% 92.95% 92.24% 91.51% 551 7,758 128,465
-@100 93.58% 95.02% 96.58% 96.20% 671 13,300 105,490
-@150 93.77% 95.24% 97.27% 96.96% 937 15,606 75,543
-@200 94.48% 95.56% 97.37% 97.07% 1,163 27,149 60,863
-@500 94.48% 95.57% 97.38% 97.09% 991 17,807 71,427
-full 94.48% 95.57% 97.38% 97.09% 862 17,354 82,116
-
-Strict accuracy — CharSoup ∩ Optimaize (63 languages, 61,811 sentences)
- ── CharSoup ── ── Optimaize ── CS(ms) Optimaize(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 82.73% 80.57% 86.30% 85.54% 471 123 131,234
-@50 93.53% 94.11% 95.40% 95.14% 490 530 126,145
-@100 95.11% 96.14% 97.23% 97.17% 630 525 98,113
-@150 95.26% 96.36% 97.33% 97.34% 745 577 82,968
-@200 96.01% 96.69% 97.34% 97.35% 1,281 978 48,252
-@500 96.02% 96.70% 97.34% 97.35% 736 566 83,982
-full 96.02% 96.70% 97.34% 97.35% 1,141 608 54,173
-
-CharSoup timing (wall-clock, full pipeline including script gate + group collapse):
-Length Wall(ms) Sent/sec
---------------------------------
-@20 1,869 68,811
-@50 2,300 55,917
-@100 2,208 58,246
-@150 2,292 56,112
-@200 3,166 40,622
-@500 4,271 30,112
-full 2,446 52,579
-
-Per-language CharSoup F1 by length:
-Language @20 @50 @100 @150 @200 @500 full
-----------------------------------------------------------------------
-ace 0.00% 0.00% 0.00% 0.00% 17.40% 17.40% 17.40%
-afr 65.70% 78.94% 82.66% 82.79% 84.09% 84.09% 84.09%
-aka 74.69% 93.08% 97.38% 97.97% 98.12% 98.12% 98.12%
-amh 93.47% 99.25% 99.75% 99.80% 99.80% 99.80% 99.80%
-ara 93.43% 99.70% 99.90% 99.90% 99.90% 99.90% 99.90%
-asm 94.92% 99.55% 99.95% 99.95% 99.95% 99.95% 99.95%
-azb 73.04% 86.02% 90.65% 91.57% 91.87% 91.81% 91.81%
-aze 82.45% 95.47% 96.93% 97.27% 97.50% 97.50% 97.50%
-bak 86.44% 98.35% 99.95% 99.95% 99.95% 99.95% 99.95%
-ban 0.00% 0.00% 0.00% 0.00% 20.04% 20.20% 20.20%
-bel 94.16% 99.70% 99.95% 99.95% 99.95% 99.95% 99.95%
-ben 94.99% 99.55% 99.95% 99.95% 99.95% 99.95% 99.95%
-bjn 0.00% 0.00% 0.00% 0.00% 16.22% 16.22% 16.22%
-bul 79.54% 97.22% 99.75% 99.85% 99.85% 99.85% 99.85%
-cat 69.32% 92.02% 96.98% 97.98% 98.18% 98.18% 98.18%
-ceb 0.00% 0.00% 0.00% 0.00% 26.31% 26.31% 26.31%
-ces 79.86% 91.96% 94.63% 94.13% 94.99% 94.99% 94.99%
-ckb 97.92% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-cym 91.23% 97.26% 98.52% 98.81% 99.06% 99.06% 99.06%
-dan 65.88% 85.85% 92.77% 94.20% 94.84% 94.84% 94.84%
-deu 69.36% 85.11% 81.83% 80.40% 82.25% 82.25% 82.25%
-ell 98.83% 99.80% 100.00% 100.00% 100.00% 100.00% 100.00%
-eng 47.86% 80.77% 93.25% 95.64% 95.85% 95.90% 95.90%
-epo 0.00% 0.00% 0.00% 0.00% 11.00% 11.00% 11.00%
-est 79.59% 95.92% 98.81% 99.20% 99.20% 99.20% 99.20%
-eus 73.46% 92.85% 97.31% 97.94% 98.13% 98.13% 98.13%
-ewe 88.48% 98.14% 99.20% 99.40% 99.45% 99.45% 99.45%
-fao 0.00% 0.00% 0.00% 0.00% 12.95% 12.95% 12.95%
-fas 72.07% 88.19% 92.05% 92.77% 92.94% 92.90% 92.90%
-fin 82.83% 97.86% 99.20% 99.55% 99.65% 99.65% 99.65%
-fra 69.74% 93.99% 98.22% 98.86% 98.96% 98.96% 98.96%
-gla 0.00% 0.00% 0.00% 0.00% 30.44% 30.44% 30.44%
-gle 62.37% 65.89% 66.44% 66.49% 70.71% 70.71% 70.71%
-glg 0.00% 0.00% 0.00% 0.00% 20.04% 20.20% 20.20%
-grn 0.00% 0.00% 0.00% 0.00% 12.77% 12.77% 12.77%
-guj 99.29% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-hau 83.81% 95.93% 95.68% 95.27% 95.45% 95.45% 95.45%
-heb 96.30% 99.65% 99.95% 99.95% 99.95% 99.95% 99.95%
-hin 60.37% 72.62% 77.02% 76.85% 77.97% 77.97% 77.97%
-hrv 72.33% 92.28% 94.83% 94.75% 95.20% 95.20% 95.20%
-hun 84.64% 96.67% 98.76% 99.01% 99.25% 99.25% 99.25%
-hye 99.60% 99.95% 99.95% 99.95% 99.95% 99.95% 99.95%
-ibo 83.21% 94.70% 97.89% 98.42% 98.52% 98.52% 98.52%
-ilo 0.00% 0.00% 0.00% 0.00% 29.43% 29.43% 29.43%
-ind 42.07% 46.99% 48.83% 48.94% 50.77% 50.79% 50.79%
-isl 65.62% 67.32% 67.03% 66.94% 68.52% 68.52% 68.52%
-ita 58.66% 83.65% 90.79% 92.01% 92.84% 92.84% 92.84%
-jav 46.24% 60.47% 63.67% 64.13% 66.82% 66.87% 66.87%
-jpn 98.34% 99.65% 99.75% 99.75% 99.75% 99.75% 99.75%
-kab 0.00% 0.00% 0.00% 0.00% 10.64% 10.64% 10.64%
-kan 99.09% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-kat 99.09% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-kaz 90.19% 99.25% 99.95% 99.95% 99.95% 99.95% 99.95%
-khm 97.69% 99.90% 99.95% 100.00% 100.00% 100.00% 100.00%
-kin 75.97% 88.03% 90.97% 90.31% 91.38% 91.38% 91.38%
-kir 84.57% 98.58% 99.95% 99.95% 99.95% 99.95% 99.95%
-kor 99.24% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-kur 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
-lao 93.94% 99.45% 99.95% 100.00% 100.00% 100.00% 100.00%
-lav 83.28% 97.78% 99.01% 99.50% 99.60% 99.60% 99.60%
-lim 0.00% 0.00% 0.00% 0.00% 14.34% 14.34% 14.34%
-lit 82.19% 96.67% 98.81% 99.06% 99.06% 99.06% 99.06%
-ltz 0.00% 0.00% 0.00% 0.00% 20.36% 20.36% 20.36%
-lug 77.05% 94.77% 97.98% 98.46% 98.61% 98.61% 98.61%
-lus 78.83% 94.70% 98.85% 99.70% 99.80% 99.80% 99.80%
-mal 98.68% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-mar 69.77% 85.31% 89.38% 89.88% 90.26% 90.26% 90.26%
-min 0.00% 0.00% 0.00% 0.00% 16.39% 16.39% 16.39%
-mkd 80.45% 96.48% 99.65% 99.80% 99.75% 99.75% 99.75%
-mlg 82.97% 96.65% 99.01% 99.40% 99.40% 99.40% 99.40%
-mlt 75.78% 89.58% 91.33% 92.39% 92.86% 92.86% 92.86%
-mon 94.85% 99.85% 100.00% 100.00% 100.00% 100.00% 100.00%
-msa 0.00% 0.00% 0.00% 0.00% 19.22% 19.54% 19.54%
-mya 98.37% 99.75% 100.00% 100.00% 100.00% 100.00% 100.00%
-nep 66.56% 83.15% 87.46% 89.02% 89.60% 89.60% 89.60%
-nld 64.64% 78.86% 78.15% 78.19% 79.25% 79.20% 79.20%
-nno 0.00% 0.00% 0.00% 0.00% 12.59% 12.59% 12.59%
-nob 52.43% 59.71% 61.56% 62.18% 63.72% 63.72% 63.72%
-nso 0.00% 0.00% 0.00% 0.00% 24.78% 24.78% 24.78%
-nya 64.36% 67.70% 67.18% 67.03% 72.15% 72.18% 72.18%
-ori 98.47% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-pan 99.09% 99.95% 99.95% 99.95% 99.95% 99.95% 99.95%
-pap 0.00% 0.00% 0.00% 0.00% 15.54% 15.54% 15.54%
-pol 66.29% 66.55% 66.80% 66.73% 68.43% 68.40% 68.40%
-por 65.25% 75.14% 75.76% 75.60% 77.96% 78.00% 78.00%
-pus 85.28% 97.20% 99.30% 99.65% 99.70% 99.70% 99.70%
-ron 82.10% 97.44% 99.25% 99.40% 99.40% 99.40% 99.40%
-rus 69.02% 95.52% 99.40% 99.50% 99.55% 99.55% 99.55%
-san 0.00% 0.00% 0.00% 0.00% 10.64% 10.64% 10.64%
-sat 99.85% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-sin 99.29% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-slk 78.55% 91.81% 94.36% 95.17% 95.26% 95.26% 95.26%
-slv 73.85% 92.61% 96.93% 97.50% 97.74% 97.74% 97.74%
-smo 78.21% 95.58% 98.57% 99.15% 99.15% 99.15% 99.15%
-sna 72.42% 90.91% 93.81% 93.65% 93.96% 93.96% 93.96%
-snd 85.16% 98.03% 99.70% 99.75% 99.75% 99.75% 99.75%
-som 67.12% 76.37% 76.02% 76.25% 77.20% 77.20% 77.20%
-spa 61.55% 75.91% 78.30% 78.73% 80.24% 80.24% 80.24%
-sqi 79.24% 94.79% 98.37% 98.71% 98.81% 98.81% 98.81%
-srp 83.42% 97.75% 99.60% 99.75% 99.80% 99.80% 99.80%
-sun 39.35% 41.97% 41.77% 41.22% 44.22% 44.25% 44.25%
-swe 68.72% 92.05% 97.45% 98.25% 98.40% 98.40% 98.40%
-swh 76.07% 93.35% 98.03% 98.08% 98.13% 98.13% 98.13%
-szl 0.00% 0.00% 0.00% 0.00% 13.82% 13.64% 13.64%
-tam 97.95% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-tat 82.94% 98.23% 99.95% 100.00% 100.00% 100.00% 100.00%
-tel 98.21% 99.65% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgk 92.43% 99.60% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgl 62.14% 76.78% 83.97% 86.95% 88.06% 88.06% 88.06%
-tha 97.69% 99.70% 100.00% 100.00% 100.00% 100.00% 100.00%
-tir 94.04% 99.50% 99.85% 99.85% 99.85% 99.85% 99.85%
-tsn 64.40% 65.59% 66.14% 66.20% 69.43% 69.45% 69.45%
-tso 74.07% 92.54% 96.82% 97.74% 97.74% 97.74% 97.74%
-tuk 78.56% 85.45% 82.72% 81.09% 81.35% 81.35% 81.35%
-tum 0.00% 0.00% 0.00% 0.00% 35.48% 35.61% 35.61%
-tur 67.72% 88.17% 91.91% 92.10% 92.57% 92.57% 92.57%
-uig 99.04% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-ukr 87.33% 99.09% 99.90% 99.90% 99.90% 99.90% 99.90%
-urd 85.83% 98.03% 99.70% 99.75% 99.65% 99.70% 99.70%
-uzb 76.20% 95.49% 98.51% 98.71% 98.96% 98.96% 98.96%
-vie 94.13% 98.42% 99.55% 99.40% 99.45% 99.45% 99.45%
-war 0.00% 0.00% 0.00% 0.00% 31.73% 31.73% 31.73%
-xho 60.78% 79.60% 87.19% 89.09% 89.53% 89.49% 89.49%
-ydd 96.71% 99.75% 99.95% 99.95% 99.95% 99.95% 99.95%
-yor 64.16% 76.83% 81.98% 83.73% 85.24% 85.24% 85.24%
-yue 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
-zho 77.88% 79.29% 79.52% 79.52% 79.52% 79.52% 79.52%
-zul 62.84% 83.88% 91.14% 91.75% 92.57% 92.52% 92.52%
-
-Per-language macro F1 (full):
-Language CharSoup OpenNLP Lingua Optimaize
-----------------------------------------------------------
-ace 17.40% N/A N/A N/A
-afr 84.09% 96.34% 96.40% 98.61%
-aka 98.12% N/A N/A N/A
-amh 99.80% 99.95% N/A N/A
-ara 99.90% 100.00% 100.00% 100.00%
-asm 99.95% 99.90% N/A N/A
-azb 91.81% N/A N/A N/A
-aze 97.50% 99.25% 98.99% N/A
-bak 99.95% 97.85% N/A N/A
-ban 20.20% 43.66% N/A N/A
-bel 99.95% 100.00% 100.00% 100.00%
-ben 99.95% 99.85% 100.00% 100.00%
-bjn 16.22% N/A N/A N/A
-bul 99.85% 98.33% 98.11% 99.14%
-cat 98.18% 98.03% 98.43% 86.74%
-ceb 26.31% 32.41% N/A N/A
-ces 94.99% 99.50% 98.79% 99.90%
-ckb 100.00% 0.00% N/A N/A
-cym 99.06% 99.95% 98.66% 99.85%
-dan 94.84% 95.45% 95.01% 98.06%
-deu 82.25% 99.30% 99.65% 99.40%
-ell 100.00% 99.95% 100.00% 100.00%
-eng 95.90% 97.22% 97.85% 98.13%
-epo 11.00% 98.75% 98.34% N/A
-est 99.20% 78.32% 99.50% 99.80%
-eus 98.13% 98.90% 99.15% 99.85%
-ewe 99.45% N/A N/A N/A
-fao 12.95% 97.72% N/A N/A
-fas 92.90% 0.59% 99.30% 99.95%
-fin 99.65% 99.39% 99.60% 99.65%
-fra 98.96% 99.25% 99.40% 99.55%
-gla 30.44% 99.55% N/A N/A
-gle 70.71% 99.50% 99.65% 99.95%
-glg 20.20% 95.11% N/A 97.22%
-grn 12.77% N/A N/A N/A
-guj 100.00% 100.00% 100.00% 100.00%
-hau 95.45% 97.59% N/A N/A
-heb 99.95% 99.85% 100.00% 99.95%
-hin 77.97% 89.24% 88.27% 99.90%
-hrv 95.20% 66.67% 68.24% 98.65%
-hun 99.25% 99.75% 99.80% 100.00%
-hye 99.95% 100.00% 100.00% N/A
-ibo 98.52% 99.55% N/A N/A
-ilo 29.43% N/A N/A N/A
-ind 50.79% 36.55% 78.31% 69.10%
-isl 68.52% 97.91% 99.70% 100.00%
-ita 92.84% 97.93% 98.30% 99.25%
-jav 66.87% 73.06% N/A N/A
-jpn 99.75% 99.65% 100.00% 68.12%
-kab 10.64% N/A N/A N/A
-kan 100.00% 100.00% N/A 100.00%
-kat 100.00% 100.00% 100.00% N/A
-kaz 99.95% 99.40% 97.61% N/A
-khm 100.00% 99.95% N/A 100.00%
-kin 91.38% 99.04% N/A N/A
-kir 99.95% 98.78% N/A N/A
-kor 100.00% 99.55% 100.00% 99.80%
-kur 0.00% 96.79% N/A N/A
-lao 100.00% 99.95% N/A N/A
-lav 99.60% 67.86% 99.30% 100.00%
-lim 14.34% 86.96% N/A N/A
-lit 99.06% 99.24% 99.50% 99.90%
-ltz 20.36% 98.99% N/A N/A
-lug 98.61% 97.64% 98.84% N/A
-lus 99.80% N/A N/A N/A
-mal 100.00% 100.00% N/A 100.00%
-mar 90.26% 97.97% 89.67% 100.00%
-min 16.39% 80.27% N/A N/A
-mkd 99.75% 97.59% 97.23% 96.98%
-mlg 99.40% 98.13% N/A N/A
-mlt 92.86% 99.34% N/A 99.90%
-mon 100.00% 99.90% 99.15% N/A
-msa 19.54% 67.94% 80.78% 40.12%
-mya 100.00% 100.00% N/A N/A
-nep 89.60% 97.56% N/A 99.90%
-nld 79.20% 90.66% 96.43% 98.53%
-nno 12.59% 88.95% 89.86% N/A
-nob 63.72% 87.81% 87.90% N/A
-nso 24.78% 95.91% N/A N/A
-nya 72.18% N/A N/A N/A
-ori 100.00% 100.00% N/A N/A
-pan 99.95% 99.95% 100.00% 100.00%
-pap 15.54% N/A N/A N/A
-pol 68.40% 99.70% 99.65% 99.95%
-por 78.00% 98.03% 98.40% 98.19%
-pus 99.70% 95.25% N/A N/A
-ron 99.40% 98.95% 97.34% 100.00%
-rus 99.55% 98.22% 98.51% 99.65%
-san 10.64% 84.87% N/A N/A
-sat 100.00% N/A N/A N/A
-sin 100.00% 100.00% N/A N/A
-slk 95.26% 99.14% 98.55% 99.70%
-slv 97.74% 97.72% 98.03% 98.60%
-smo 99.15% N/A N/A N/A
-sna 93.96% N/A 99.00% N/A
-snd 99.75% 99.95% N/A N/A
-som 77.20% 99.60% 99.80% 99.95%
-spa 80.24% 84.02% 98.05% 92.82%
-sqi 98.81% 99.70% 99.70% 100.00%
-srp 99.80% 99.14% 97.61% 97.28%
-sun 44.25% N/A N/A N/A
-swe 98.40% 98.42% 98.45% 99.50%
-swh 98.13% N/A N/A N/A
-szl 13.64% N/A N/A N/A
-tam 100.00% 100.00% 100.00% 100.00%
-tat 100.00% 97.72% N/A N/A
-tel 100.00% 99.55% 99.95% 100.00%
-tgk 100.00% 100.00% N/A N/A
-tgl 88.06% 54.09% 96.87% 99.65%
-tha 100.00% 99.55% 97.90% 100.00%
-tir 99.85% N/A N/A N/A
-tsn 69.45% 95.18% 92.46% N/A
-tso 97.74% N/A 98.90% N/A
-tuk 81.35% 99.75% N/A N/A
-tum 35.61% N/A N/A N/A
-tur 92.57% 98.53% 98.85% 99.95%
-uig 100.00% 91.88% N/A N/A
-ukr 99.90% 99.60% 97.86% 99.85%
-urd 99.70% 96.75% 99.30% 100.00%
-uzb 98.96% 98.21% N/A N/A
-vie 99.45% 99.90% 99.05% 100.00%
-war 31.73% 11.58% N/A N/A
-xho 89.49% 84.42% 90.94% N/A
-ydd 99.95% N/A N/A N/A
-yor 85.24% 98.58% 97.38% N/A
-yue 0.00% N/A N/A N/A
-zho 79.52% N/A 100.00% 87.76%
-zul 92.52% 81.80% 91.63% N/A
-
-CharSoup top confusions (languages with F1 < 95%, @20):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-ace 0.0% sun→454, unk→107, ind→73, jav→53, ibo→23, tgl→22, kha→17
-ban 0.0% jav→359, sun→125, ind→124, unk→57, tgl→44, sna→22, tet→19
-bjn 0.0% sun→226, ind→189, jav→160, tgl→55, unk→45, tet→31, trv→30
-ceb 0.0% hil→565, tgl→300, unk→24, trv→12, sqi→10, dag→7, tay→7
-epo 0.0% som→265, unk→153, yor→73, slv→43, tet→40, spa→40, lit→34
-fao 0.0% isl→725, unk→54, nob→43, swe→17, ita→12, aze→11, hun→10
-gla 0.0% gle→862, unk→30, lus→17, mlg→9, vie→8, smo→6, yor→6
-glg 0.0% por→325, spa→152, unk→90, tet→58, cat→47, ita→43, eng→37
-grn 0.0% unk→269, eus→74, tet→55, ita→51, slk→49, yor→39, spa→36
-ilo 0.0% sun→237, tgl→119, yor→83, unk→71, som→60, hil→37, dag→32
-kab 0.0% unk→336, som→49, xho→41, aze→36, kin→27, aka→27, cym→25
-kur 0.0% unk→327, tuk→50, yor→47, dag→44, ita→40, tur→30, som→25
-lim 0.0% afr→172, unk→145, nld→143, fry→130, kin→36, fra→29, deu→28
-ltz 0.0% unk→279, deu→107, fry→84, gle→41, hyw→35, afr→29, fra→28
-min 0.0% ind→226, jav→122, sun→119, tet→66, unk→59, tgl→56, eus→45
-msa 0.0% ind→651, sun→87, jav→70, tet→24, unk→18, smo→13, sna→10
-nno 0.0% nob→487, unk→84, swe→74, dan→61, isl→51, afr→19, sqi→18
-nso 0.0% tsn→538, unk→57, tso→40, smo→27, yor→22, dag→19, tet→19
-pap 0.0% unk→129, yor→96, ita→95, mlt→81, tet→60, sun→58, som→39
-san 0.0% hin→554, mar→253, nep→174, unk→13, sna→1, cat→1, eng→1
-szl 0.0% pol→656, unk→73, yor→39, ces→24, slk→21, hrv→20, hun→19
-tum 0.0% nya→584, swh→73, sna→60, kin→30, xho→30, lug→28, yor→27
-war 0.0% hil→401, tgl→101, tet→68, trv→49, unk→41, som→37, tso→33
-yue 0.0% zho→949, unk→33, jpn→5, gle→1, fra→1, spa→1, ita→1
-sun 39.3% jav→67, ind→57, unk→32, mlg→18, tgl→15, tet→14, som→12
-ind 42.1% sun→73, jav→54, tet→39, unk→25, trv→12, smo→10, tgl→8
-jav 46.2% sun→67, ind→57, unk→33, mlg→15, tgl→14, trv→13, sqi→13
-eng 47.9% unk→158, dag→33, ita→29, mlg→22, yor→17, aze→17, orm→17
-nob 52.4% dan→96, unk→70, swe→30, fry→20, afr→16, isl→15, ita→13
-ita 58.7% cat→35, unk→32, mlt→20, smo→16, fra→13, som→10, tet→9
-hin 60.4% mar→62, nep→55, unk→3, spa→1
-xho 60.8% zul→89, unk→23, yor→21, lug→13, sna→9, tso→7, kin→6
-spa 61.6% cat→86, unk→34, por→28, tet→27, ita→22, yor→14, eus→13
-tgl 62.1% hil→79, trv→8, sun→6, ibo→5, lus→4, som→4, unk→4
-gle 62.4% unk→8, eng→5, lus→4, spa→2, sun→2, ron→2, smo→2
-zul 62.8% xho→317, yor→45, lug→21, unk→14, kin→6, hrv→6, nya→5
-yor 64.2% dag→11, unk→8, eng→5, aka→5, smo→3, tet→3, som→2
-nya 64.4% unk→17, lug→15, xho→14, swh→13, sna→11, tso→9, smo→6
-tsn 64.4% smo→34, unk→20, tet→16, lug→14, swh→13, ita→12, tgl→12
-nld 64.6% afr→119, fry→108, unk→23, eng→22, deu→14, yor→12, kin→11
-por 65.2% unk→43, tet→37, cat→21, ita→19, lit→18, eng→16, spa→16
-isl 65.6% unk→16, eng→6, nob→4, ita→4, hun→3, tso→3, lit→3
-afr 65.7% kin→46, nld→25, unk→24, fry→18, eng→17, som→8, lav→4
-dan 65.9% nob→117, unk→53, swe→31, fry→25, afr→21, isl→11, ita→10
-pol 66.3% unk→18, ces→9, hrv→9, slk→9, slv→8, ita→6, xho→5
-nep 66.6% hin→270, mar→85, unk→9
-som 67.1% unk→9, orm→7, est→3, smo→3, sna→3, xho→3, dag→2
-tur 67.7% aze→69, tuk→66, unk→45, sna→31, ita→11, uzb→6, sqi→5
-swe 68.7% unk→70, nob→48, dan→28, fin→20, som→10, ita→10, sna→10
-rus 69.0% udm→51, bul→46, srp→43, che→42, chv→34, ukr→28, tgk→25
-cat 69.3% unk→32, ita→21, spa→20, por→12, fra→11, lav→10, mlg→10
-deu 69.4% unk→61, afr→60, kin→16, ita→13, lav→13, fry→12, gle→10
-fra 69.7% unk→69, cat→42, smo→21, yor→19, mlg→19, eng→15, dag→14
-mar 69.8% hin→205, nep→43
-fas 72.1% pus→92, snd→91, azb→51, urd→13, ckb→10, unk→4, som→2
-hrv 72.3% slv→127, unk→22, yor→20, slk→14, eus→8, ces→8, ita→6
-sna 72.4% nya→28, unk→20, xho→18, lug→12, smo→8, swh→6, tso→4
-azb 73.0% fas→228, pus→44, urd→39, snd→19, ckb→15, unk→3
-eus 73.5% unk→48, tet→23, ita→15, afr→8, yor→6, slv→6, pol→6
-slv 73.9% hrv→59, yor→32, slk→12, ces→10, unk→10, ita→5, sqi→3
-tso 74.1% sna→23, nya→23, xho→16, sun→15, unk→15, lug→15, yor→14
-aka 74.7% unk→55, dag→20, ibo→19, eng→11, lug→9, som→7, smo→6
-mlt 75.8% unk→64, ita→35, som→17, dag→9, yor→8, ron→8, eng→8
-kin 76.0% xho→23, swh→12, yor→12, sna→12, unk→12, lug→12, zul→7
-swh 76.1% sna→19, dag→15, yor→13, som→12, unk→12, nya→12, smo→11
-uzb 76.2% unk→39, aze→30, tuk→28, eus→13, yor→12, tur→11, som→10
-lug 77.1% xho→24, kin→21, unk→12, swh→8, sna→8, nya→8, som→6
-zho 77.9% unk→97, jpn→9, spa→4, ita→2, cat→1, tgl→1, tsn→1
-smo 78.2% ibo→11, unk→10, gle→3, tet→3, mlg→3, yor→3, sqi→2
-slk 78.6% ces→49, slv→17, yor→12, unk→9, hrv→8, mlg→4, sqi→3
-tuk 78.6% tur→40, unk→37, aze→9, ind→5, yor→4, pol→3, ita→3
-lus 78.8% unk→50, tso→36, eng→32, yor→9, aka→8, xho→8, dag→7
-sqi 79.2% unk→18, slv→9, ita→7, som→5, yor→5, hrv→4, swh→3
-bul 79.5% mkd→74, srp→28, rus→21, tgk→19, ukr→9, sah→8, chv→6
-est 79.6% unk→39, fin→19, som→8, smo→8, lit→7, eng→6, tet→6
-ces 79.9% slk→101, slv→18, unk→17, yor→14, pol→8, hrv→8, est→4
-mkd 80.5% bul→65, srp→57, tgk→12, che→9, rus→8, udm→6, bak→5
-ron 82.1% unk→22, ita→15, cat→13, smo→11, fra→9, tet→7, tur→5
-lit 82.2% unk→36, lav→19, smo→9, ita→8, slv→7, est→5, cat→4
-aze 82.5% tur→43, unk→14, tuk→7, uzb→6, eus→4, swh→3, ita→3
-fin 82.8% est→55, unk→26, ita→16, som→11, smo→8, deu→5, lit→4
-tat 82.9% bak→66, kir→30, rus→21, sah→19, kaz→15, udm→15, unk→14
-mlg 83.0% smo→7, tet→5, unk→4, spa→2, hrv→2, tso→2, aka→2
-ibo 83.2% yor→9, unk→8, smo→4, nya→4, swh→3, hil→3, som→3
-lav 83.3% unk→20, lit→16, hrv→5, est→4, cym→4, fin→3, slv→3
-srp 83.4% mkd→95, bul→23, rus→8, tgk→6, che→4, kir→3, ukr→3
-hau 83.8% swh→13, unk→12, som→9, trv→9, dag→8, sna→7, ita→6
-kir 84.6% kaz→27, rus→27, sah→21, tat→16, tgk→16, mon→15, unk→14
-hun 84.6% unk→35, slk→7, ita→6, eus→5, cat→4, eng→4, tet→4
-snd 85.2% pus→10, ara→6, urd→6, azb→5, fas→4, unk→2, heb→1
-pus 85.3% azb→38, fas→33, unk→17, snd→13, urd→8, ckb→5, som→2
-urd 85.8% snd→135, fas→17, azb→16, unk→2, ckb→2, som→1, spa→1
-bak 86.4% tat→37, kaz→24, rus→12, bul→12, unk→10, kir→8, sah→6
-ukr 87.3% srp→35, rus→25, bul→23, bel→16, tgk→15, mkd→12, chv→8
-ewe 88.5% unk→35, aka→10, yor→8, ibo→6, eus→4, dag→4, smo→4
-kaz 90.2% kir→20, unk→18, bak→13, rus→9, tgk→9, bul→8, ukr→6
-cym 91.2% gle→8, unk→6, pol→3, ita→3, lav→2, spa→1, mlg→1
-tgk 92.4% che→7, bul→7, mon→3, unk→3, udm→3, mkd→2, srp→2
-ara 93.4% snd→39, urd→27, pus→26, fas→14, azb→4, unk→2, ckb→2
-amh 93.5% tir→16, unk→9, orm→3, eng→3, lus→2, tet→1, spa→1
-lao 93.9% unk→99, ita→3, spa→2, sna→2, xho→2, eng→2, gle→1
-tir 94.0% amh→96, lav→1, unk→1
-vie 94.1% unk→12, xho→3, gle→2, kha→2, spa→2, hil→2, lus→2
-bel 94.2% rus→14, ukr→11, mkd→5, sah→5, udm→5, che→4, mon→3
-mon 94.8% sah→19, che→8, tgk→6, kir→5, rus→4, oss→4, udm→3
-asm 94.9% ben→52, unk→2
-ben 95.0% asm→47, unk→1
-
-CharSoup top confusions (languages with F1 < 95%, full):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-kur 0.0% tuk→315, tur→70, yor→70, diq→59, hau→49, trv→46, kin→41
-yue 0.0% zho→980, ita→4, ces→2, unk→2, som→1, fra→1, sna→1
-kab 10.6% tuk→132, kin→112, som→89, tur→69, nob→68, xho→58, mlt→48
-san 10.6% hin→531, mar→207, nep→203
-epo 11.0% som→452, tet→102, yor→98, spa→76, hrv→66, slv→36, ita→13
-nno 12.6% nob→916, dan→11, swe→1, ita→1, hrv→1
-grn 12.8% tet→302, slk→85, spa→81, sna→66, ces→65, ita→49, yor→39
-fao 12.9% isl→915, nob→8, dan→1, ces→1, cat→1, uzb→1, hrv→1
-szl 13.6% pol→917, slk→2, ces→1, ewe→1, ita→1, tuk→1, hrv→1
-lim 14.3% nld→436, afr→324, fry→135, kin→6, eng→4, por→3, deu→3
-pap 15.5% tet→411, sun→189, mlt→78, yor→65, ita→46, tgl→23, ind→16
-bjn 16.2% sun→503, ind→290, jav→109, tgl→4, hil→2, tet→1
-min 16.4% ind→507, sun→197, jav→128, tet→43, hil→22, tgl→5, tur→1
-ace 17.4% sun→862, ind→24, jav→9, ita→2, tgl→1, mlg→1, hil→1
-msa 19.5% ind→870, sun→15, tet→1, jav→1
-ban 20.2% jav→678, sun→126, ind→70, tet→2, tgl→2, hil→2, smo→1
-glg 20.2% por→519, spa→320, tet→36, cat→4, tur→1, gle→1, fra→1
-ltz 20.4% deu→420, fry→173, nob→71, nld→48, dan→31, afr→29, ces→18
-nso 24.8% tsn→850, tso→2, som→1, tet→1, swh→1, dag→1
-ceb 26.3% hil→801, tgl→45
-ilo 29.4% sun→484, tgl→177, hil→98, yor→19, tet→14, aka→9, som→7
-gla 30.4% gle→818
-war 31.7% hil→765, tet→11, tso→10, tgl→6, trv→5, sun→4, ita→2
-tum 35.6% nya→753, swh→13, tso→4, sna→4, kin→3, zul→2, trv→1
-sun 44.3% ind→4, jav→2, ita→1, hil→1
-ind 50.8% msa→19, sun→13, jav→3
-nob 63.7% dan→16, swe→1
-jav 66.9% sun→4, ind→3, hau→1
-pol 68.4% (no misses recorded)
-isl 68.5% (no misses recorded)
-tsn 69.5% (no misses recorded)
-gle 70.7% (no misses recorded)
-nya 72.2% tso→1, yor→1
-som 77.2% (no misses recorded)
-hin 78.0% nep→8
-por 78.0% tet→1
-nld 79.2% afr→9, fry→9, vls→4, deu→1, slv→1
-zho 79.5% unk→7, eng→3, spa→2, jpn→2, tsn→2, ita→2, hun→1
-spa 80.2% por→1, cat→1
-tuk 81.4% (no misses recorded)
-deu 82.2% afr→1
-afr 84.1% nld→1
-yor 85.2% ibo→1
-tgl 88.1% hil→1
-xho 89.5% zul→62, kin→1, sna→1, yor→1
-nep 89.6% hin→14, mar→1
-mar 90.3% hin→4, nep→2
-kin 91.4% (no misses recorded)
-azb 91.8% fas→149, pus→2
-zul 92.5% xho→35, som→1, sna→1, yor→1, lug→1
-tur 92.6% sna→2, tuk→2, aze→2
-ita 92.8% mlt→4, smo→1
-mlt 92.9% ind→1, jav→1
-fas 92.9% pus→2, eng→1
-sna 94.0% tet→1, swh→1
-dan 94.8% nob→12, sna→1
-ces 95.0% slk→1
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/flores-SHORT_TEXT.log b/docs/modules/ROOT/pages/advanced/lang-detection/flores-SHORT_TEXT.log
deleted file mode 100644
index f35be0f286..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/flores-SHORT_TEXT.log
+++ /dev/null
@@ -1,414 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-=== Language Detection Comparison Report ===
-
-Test sentences: 203,381
-CharSoup ∩ OpenNLP: 91 languages, 90,726 sentences
-CharSoup ∩ Lingua: 67 languages, 67,793 sentences
-CharSoup ∩ Optimaize: 59 languages, 59,817 sentences
-
-Model heap (approx):
- CharSoup: ~0.0 MB
- OpenNLP: ~75.3 MB
- Lingua: ~0.1 MB (low accuracy mode)
- Optimaize: ~95.0 MB
-
-Coverage-adjusted accuracy — each detector scored on its own supported languages only
- (test sentences whose true language is not in a detector's covered set are skipped)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─ CS(ms) ON(ms) Li(ms) Opt(ms) CS sent/s
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------------------------------------------------------
-@20 84.68% 83.59% 74.87% 72.85% 76.35% 75.88% 84.87% 84.38% 1,909 1,795 13,067 646 55,358
-@50 96.06% 96.31% 86.09% 85.43% 90.99% 90.63% 94.44% 94.29% 2,047 1,379 24,971 3,951 51,626
-@100 97.92% 98.29% 90.25% 90.12% 95.43% 95.33% 96.51% 96.50% 2,561 2,591 43,979 4,106 41,264
-@150 98.11% 98.50% 90.98% 90.97% 96.15% 96.11% 96.72% 96.76% 3,008 4,034 52,015 4,256 35,132
-@200 98.14% 98.52% 91.11% 91.11% 96.23% 96.20% 96.75% 96.79% 2,926 4,098 55,078 4,280 36,117
-@500 98.14% 98.52% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 3,007 4,257 55,677 3,701 35,144
-full 98.14% 98.52% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 2,589 3,707 98,812 4,170 40,818
-
-Breadth-weighted accuracy — all 203 FLORES languages, unsupported languages score 0
- (penalises limited coverage; use this to compare total useful output across all inputs)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------
-@20 43.80% 43.43% 42.05% 40.71% 27.46% 27.53% 26.76% 26.89%
-@50 49.68% 50.04% 48.35% 47.74% 32.72% 32.88% 29.77% 30.04%
-@100 50.65% 51.07% 50.68% 50.36% 34.32% 34.58% 30.43% 30.75%
-@150 50.75% 51.18% 51.09% 50.84% 34.58% 34.86% 30.49% 30.83%
-@200 50.76% 51.19% 51.16% 50.91% 34.61% 34.90% 30.50% 30.84%
-@500 50.76% 51.19% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-full 50.76% 51.19% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-
-Strict accuracy — CharSoup ∩ OpenNLP (91 languages, 90,726 sentences)
- ── CharSoup ── ── OpenNLP ── CS(ms) OpenNLP(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 85.20% 83.43% 81.12% 76.89% 800 400 113,408
-@50 96.25% 96.37% 91.01% 89.01% 841 636 107,879
-@100 97.95% 98.29% 94.28% 93.26% 1,144 1,129 79,306
-@150 98.11% 98.49% 94.88% 94.03% 1,252 1,619 72,465
-@200 98.14% 98.52% 95.01% 94.16% 1,318 1,959 68,836
-@500 98.14% 98.51% 95.02% 94.18% 1,127 1,702 80,502
-full 98.14% 98.51% 95.02% 94.18% 1,165 1,733 77,876
-
-Strict accuracy — CharSoup ∩ Lingua (67 languages, 67,793 sentences)
- ── CharSoup ── ── Lingua ── CS(ms) Lingua(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 85.77% 83.05% 79.29% 77.69% 494 3,575 137,233
-@50 97.48% 97.05% 93.17% 92.02% 601 7,614 112,800
-@100 99.34% 99.21% 97.21% 96.56% 835 12,979 81,189
-@150 99.53% 99.44% 97.83% 97.28% 954 15,515 71,062
-@200 99.56% 99.47% 97.92% 97.39% 966 16,789 70,179
-@500 99.56% 99.47% 97.92% 97.40% 909 15,986 74,580
-full 99.56% 99.47% 97.92% 97.40% 921 16,150 73,608
-
-Strict accuracy — CharSoup ∩ Optimaize (59 languages, 59,817 sentences)
- ── CharSoup ── ── Optimaize ── CS(ms) Optimaize(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 86.44% 83.25% 87.83% 86.65% 461 128 129,755
-@50 97.85% 97.25% 96.81% 96.25% 566 535 105,684
-@100 99.54% 99.35% 98.55% 98.25% 749 534 79,862
-@150 99.70% 99.57% 98.74% 98.50% 858 554 69,717
-@200 99.72% 99.60% 98.77% 98.53% 942 570 63,500
-@500 99.72% 99.60% 98.77% 98.53% 724 488 82,620
-full 99.72% 99.60% 98.77% 98.53% 744 522 80,399
-
-CharSoup timing (wall-clock, full pipeline including script gate + group collapse):
-Length Wall(ms) Sent/sec
---------------------------------
-@20 1,909 55,358
-@50 2,047 51,626
-@100 2,561 41,264
-@150 3,008 35,132
-@200 2,926 36,117
-@500 3,007 35,144
-full 2,589 40,818
-
-Per-language CharSoup F1 by length:
-Language @20 @50 @100 @150 @200 @500 full
-----------------------------------------------------------------------
-afr 74.34% 93.79% 98.61% 99.06% 99.10% 99.10% 99.10%
-aka 79.98% 95.90% 99.30% 99.60% 99.60% 99.60% 99.60%
-amh 93.47% 99.25% 99.75% 99.80% 99.80% 99.80% 99.80%
-ara 93.43% 99.70% 99.90% 99.90% 99.90% 99.90% 99.90%
-asm 94.92% 99.55% 99.95% 99.95% 99.95% 99.95% 99.95%
-azb 73.04% 86.02% 90.65% 91.57% 91.81% 91.81% 91.81%
-aze 85.38% 97.32% 98.47% 98.66% 98.76% 98.71% 98.71%
-bak 86.44% 98.35% 99.95% 99.95% 99.95% 99.95% 99.95%
-bel 94.16% 99.70% 99.95% 99.95% 99.95% 99.95% 99.95%
-ben 94.99% 99.55% 99.95% 99.95% 99.95% 99.95% 99.95%
-bul 79.54% 97.22% 99.75% 99.85% 99.90% 99.90% 99.90%
-cat 74.46% 96.15% 99.50% 99.80% 99.85% 99.85% 99.85%
-ces 82.86% 96.41% 99.25% 99.10% 99.25% 99.25% 99.25%
-ckb 97.92% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-cym 94.55% 99.55% 99.80% 99.90% 99.90% 99.90% 99.90%
-dan 69.67% 91.46% 97.35% 98.30% 98.40% 98.40% 98.40%
-deu 76.37% 97.31% 99.45% 99.70% 99.75% 99.75% 99.75%
-ell 98.83% 99.80% 100.00% 100.00% 100.00% 100.00% 100.00%
-eng 52.43% 83.61% 94.37% 96.38% 96.59% 96.59% 96.59%
-est 83.57% 98.46% 99.90% 99.90% 99.90% 99.90% 99.90%
-eus 82.49% 98.74% 99.95% 100.00% 100.00% 100.00% 100.00%
-ewe 90.17% 98.54% 99.25% 99.45% 99.50% 99.50% 99.50%
-fas 72.07% 88.19% 92.05% 92.77% 92.94% 92.94% 92.94%
-fin 84.94% 98.99% 99.95% 100.00% 100.00% 100.00% 100.00%
-fra 74.83% 96.63% 99.30% 99.65% 99.65% 99.65% 99.65%
-gle 92.75% 99.45% 99.85% 99.95% 99.95% 99.95% 99.95%
-guj 99.29% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-hau 89.38% 98.36% 97.84% 97.51% 97.36% 97.36% 97.36%
-heb 96.30% 99.65% 99.95% 99.95% 99.95% 99.95% 99.95%
-hin 74.62% 91.18% 97.73% 98.56% 98.65% 98.65% 98.65%
-hrv 76.57% 96.49% 99.05% 99.15% 99.20% 99.20% 99.20%
-hun 89.25% 99.20% 99.90% 99.90% 99.90% 99.90% 99.90%
-hye 99.60% 99.95% 99.95% 99.95% 99.95% 99.95% 99.95%
-ibo 89.36% 97.88% 99.30% 99.65% 99.75% 99.70% 99.70%
-ind 72.18% 91.19% 96.01% 97.22% 97.22% 97.27% 97.27%
-isl 92.67% 99.50% 99.95% 100.00% 100.00% 100.00% 100.00%
-ita 68.98% 93.53% 98.40% 99.00% 99.20% 99.20% 99.20%
-jav 68.43% 93.78% 98.40% 98.85% 99.00% 99.00% 99.00%
-jpn 98.59% 99.70% 99.80% 99.80% 99.80% 99.80% 99.80%
-kan 99.09% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-kat 99.09% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-kaz 90.19% 99.25% 99.95% 99.95% 99.95% 99.95% 99.95%
-khm 97.69% 99.90% 99.95% 100.00% 100.00% 100.00% 100.00%
-kin 83.48% 96.52% 98.08% 97.65% 97.75% 97.75% 97.75%
-kir 84.57% 98.58% 99.95% 99.95% 99.95% 99.95% 99.95%
-kor 99.24% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-kur 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
-lao 93.94% 99.45% 99.95% 100.00% 100.00% 100.00% 100.00%
-lav 88.57% 99.25% 99.90% 99.95% 100.00% 100.00% 100.00%
-lit 87.30% 99.30% 99.95% 99.95% 99.95% 99.95% 99.95%
-lug 81.92% 96.98% 99.15% 99.15% 99.25% 99.25% 99.25%
-lus 82.93% 95.44% 99.14% 99.70% 99.75% 99.75% 99.75%
-mal 98.68% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-mar 79.13% 95.87% 99.24% 99.60% 99.65% 99.65% 99.65%
-mkd 80.45% 96.48% 99.65% 99.80% 99.80% 99.80% 99.80%
-mlg 89.97% 98.51% 99.65% 99.75% 99.80% 99.80% 99.80%
-mlt 81.53% 97.80% 99.35% 99.40% 99.50% 99.50% 99.50%
-mon 94.85% 99.85% 100.00% 100.00% 100.00% 100.00% 100.00%
-mya 98.37% 99.75% 100.00% 100.00% 100.00% 100.00% 100.00%
-nep 73.26% 92.45% 97.87% 98.69% 98.74% 98.74% 98.74%
-nld 71.08% 92.68% 97.75% 98.78% 98.89% 98.89% 98.89%
-nob 69.11% 90.04% 96.47% 97.95% 98.00% 98.00% 98.00%
-nya 85.18% 97.42% 99.50% 99.75% 99.65% 99.70% 99.70%
-ori 98.47% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-pan 99.09% 99.95% 99.95% 99.95% 99.95% 99.95% 99.95%
-pol 90.19% 98.56% 99.95% 99.95% 99.95% 99.95% 99.95%
-por 79.56% 97.42% 99.65% 99.75% 99.75% 99.75% 99.75%
-pus 85.28% 97.20% 99.30% 99.65% 99.65% 99.65% 99.65%
-ron 86.39% 99.00% 99.95% 99.90% 99.90% 99.90% 99.90%
-rus 69.02% 95.52% 99.40% 99.50% 99.55% 99.55% 99.55%
-sat 99.85% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-sin 99.29% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-slk 83.22% 97.47% 99.70% 99.80% 99.85% 99.85% 99.85%
-slv 78.15% 96.35% 99.40% 99.60% 99.60% 99.60% 99.60%
-smo 85.53% 98.42% 99.50% 99.85% 99.85% 99.85% 99.85%
-sna 80.19% 96.22% 97.69% 97.64% 97.64% 97.69% 97.69%
-snd 85.16% 98.03% 99.70% 99.75% 99.75% 99.75% 99.75%
-som 85.20% 96.98% 97.84% 98.66% 98.62% 98.66% 98.66%
-spa 71.59% 95.40% 99.35% 99.70% 99.75% 99.75% 99.75%
-sqi 88.15% 99.35% 99.85% 99.95% 99.95% 99.95% 99.95%
-srp 83.42% 97.75% 99.60% 99.75% 99.80% 99.80% 99.80%
-sun 68.89% 89.93% 96.94% 97.72% 97.82% 97.87% 97.87%
-swe 73.91% 95.19% 98.89% 99.24% 99.24% 99.24% 99.24%
-swh 81.91% 97.11% 99.50% 99.50% 99.50% 99.50% 99.50%
-tam 97.95% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-tat 82.94% 98.23% 99.95% 100.00% 100.00% 100.00% 100.00%
-tel 98.21% 99.65% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgk 92.43% 99.60% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgl 86.19% 97.69% 99.25% 99.70% 99.70% 99.70% 99.70%
-tha 97.69% 99.70% 100.00% 100.00% 100.00% 100.00% 100.00%
-tir 94.04% 99.50% 99.85% 99.85% 99.85% 99.85% 99.85%
-tsn 84.23% 95.93% 98.47% 98.86% 98.91% 98.91% 98.91%
-tso 80.10% 96.05% 98.31% 98.86% 98.76% 98.76% 98.76%
-tuk 82.16% 89.99% 87.56% 85.99% 85.36% 85.21% 85.21%
-tur 71.66% 93.15% 96.25% 96.21% 96.12% 96.17% 96.17%
-uig 99.04% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-ukr 87.33% 99.09% 99.90% 99.90% 99.90% 99.90% 99.90%
-urd 85.83% 98.03% 99.70% 99.75% 99.75% 99.75% 99.75%
-uzb 80.28% 97.45% 99.45% 99.35% 99.50% 99.45% 99.45%
-vie 95.30% 98.56% 99.55% 99.40% 99.45% 99.45% 99.45%
-xho 65.79% 84.82% 92.12% 93.26% 93.57% 93.51% 93.51%
-ydd 96.71% 99.75% 99.95% 99.95% 99.95% 99.95% 99.95%
-yor 77.81% 93.36% 95.63% 95.82% 95.91% 95.91% 95.91%
-zho 97.00% 98.99% 99.24% 99.24% 99.24% 99.24% 99.24%
-zul 63.94% 85.37% 92.29% 93.09% 93.47% 93.42% 93.42%
-
-Per-language macro F1 (full):
-Language CharSoup OpenNLP Lingua Optimaize
-----------------------------------------------------------
-afr 99.10% 96.34% 96.40% 98.61%
-aka 99.60% N/A N/A N/A
-amh 99.80% 99.95% N/A N/A
-ara 99.90% 100.00% 100.00% 100.00%
-asm 99.95% 99.90% N/A N/A
-azb 91.81% N/A N/A N/A
-aze 98.71% 99.25% 98.99% N/A
-bak 99.95% 97.85% N/A N/A
-bel 99.95% 100.00% 100.00% 100.00%
-ben 99.95% 99.85% 100.00% 100.00%
-bul 99.90% 98.33% 98.11% 99.14%
-cat 99.85% 98.03% 98.43% 86.74%
-ces 99.25% 99.50% 98.79% 99.90%
-ckb 100.00% 0.00% N/A N/A
-cym 99.90% 99.95% 98.66% 99.85%
-dan 98.40% 95.45% 95.01% 98.06%
-deu 99.75% 99.30% 99.65% 99.40%
-ell 100.00% 99.95% 100.00% 100.00%
-eng 96.59% 97.22% 97.85% 98.13%
-est 99.90% 78.32% 99.50% 99.80%
-eus 100.00% 98.90% 99.15% 99.85%
-ewe 99.50% N/A N/A N/A
-fas 92.94% 0.59% 99.30% 99.95%
-fin 100.00% 99.39% 99.60% 99.65%
-fra 99.65% 99.25% 99.40% 99.55%
-gle 99.95% 99.50% 99.65% 99.95%
-guj 100.00% 100.00% 100.00% 100.00%
-hau 97.36% 97.59% N/A N/A
-heb 99.95% 99.85% 100.00% 99.95%
-hin 98.65% 89.24% 88.27% 99.90%
-hrv 99.20% 66.67% 68.24% 98.65%
-hun 99.90% 99.75% 99.80% 100.00%
-hye 99.95% 100.00% 100.00% N/A
-ibo 99.70% 99.55% N/A N/A
-ind 97.27% 36.55% 78.31% 69.10%
-isl 100.00% 97.91% 99.70% 100.00%
-ita 99.20% 97.93% 98.30% 99.25%
-jav 99.00% 73.06% N/A N/A
-jpn 99.80% 99.65% 100.00% 68.12%
-kan 100.00% 100.00% N/A 100.00%
-kat 100.00% 100.00% 100.00% N/A
-kaz 99.95% 99.40% 97.61% N/A
-khm 100.00% 99.95% N/A 100.00%
-kin 97.75% 99.04% N/A N/A
-kir 99.95% 98.78% N/A N/A
-kor 100.00% 99.55% 100.00% 99.80%
-kur 0.00% 96.79% N/A N/A
-lao 100.00% 99.95% N/A N/A
-lav 100.00% 67.86% 99.30% 100.00%
-lit 99.95% 99.24% 99.50% 99.90%
-lug 99.25% 97.64% 98.84% N/A
-lus 99.75% N/A N/A N/A
-mal 100.00% 100.00% N/A 100.00%
-mar 99.65% 97.97% 89.67% 100.00%
-mkd 99.80% 97.59% 97.23% 96.98%
-mlg 99.80% 98.13% N/A N/A
-mlt 99.50% 99.34% N/A 99.90%
-mon 100.00% 99.90% 99.15% N/A
-mya 100.00% 100.00% N/A N/A
-nep 98.74% 97.56% N/A 99.90%
-nld 98.89% 90.66% 96.43% 98.53%
-nob 98.00% 87.81% 87.90% N/A
-nya 99.70% N/A N/A N/A
-ori 100.00% 100.00% N/A N/A
-pan 99.95% 99.95% 100.00% 100.00%
-pol 99.95% 99.70% 99.65% 99.95%
-por 99.75% 98.03% 98.40% 98.19%
-pus 99.65% 95.25% N/A N/A
-ron 99.90% 98.95% 97.34% 100.00%
-rus 99.55% 98.22% 98.51% 99.65%
-sat 100.00% N/A N/A N/A
-sin 100.00% 100.00% N/A N/A
-slk 99.85% 99.14% 98.55% 99.70%
-slv 99.60% 97.72% 98.03% 98.60%
-smo 99.85% N/A N/A N/A
-sna 97.69% N/A 99.00% N/A
-snd 99.75% 99.95% N/A N/A
-som 98.66% 99.60% 99.80% 99.95%
-spa 99.75% 84.02% 98.05% 92.82%
-sqi 99.95% 99.70% 99.70% 100.00%
-srp 99.80% 99.14% 97.61% 97.28%
-sun 97.87% N/A N/A N/A
-swe 99.24% 98.42% 98.45% 99.50%
-swh 99.50% N/A N/A N/A
-tam 100.00% 100.00% 100.00% 100.00%
-tat 100.00% 97.72% N/A N/A
-tel 100.00% 99.55% 99.95% 100.00%
-tgk 100.00% 100.00% N/A N/A
-tgl 99.70% 54.09% 96.87% 99.65%
-tha 100.00% 99.55% 97.90% 100.00%
-tir 99.85% N/A N/A N/A
-tsn 98.91% 95.18% 92.46% N/A
-tso 98.76% N/A 98.90% N/A
-tuk 85.21% 99.75% N/A N/A
-tur 96.17% 98.53% 98.85% 99.95%
-uig 100.00% 91.88% N/A N/A
-ukr 99.90% 99.60% 97.86% 99.85%
-urd 99.75% 96.75% 99.30% 100.00%
-uzb 99.45% 98.21% N/A N/A
-vie 99.45% 99.90% 99.05% 100.00%
-xho 93.51% 84.42% 90.94% N/A
-ydd 99.95% N/A N/A N/A
-yor 95.91% 98.58% 97.38% N/A
-zho 99.24% N/A 100.00% 87.76%
-zul 93.42% 81.80% 91.63% N/A
-
-CharSoup top confusions (languages with F1 < 95%, @20):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-kur 0.0% unk→327, tuk→50, yor→47, dag→44, ita→40, tur→30, som→25
-eng 52.4% unk→158, dag→33, ita→29, mlg→22, yor→17, aze→17, orm→17
-zul 63.9% xho→317, yor→45, lug→21, unk→14, kin→6, hrv→6, nya→5
-xho 65.8% zul→89, unk→23, yor→21, lug→13, sna→9, tso→7, kin→6
-jav 68.4% sun→67, ind→57, unk→33, mlg→15, tgl→14, trv→13, sqi→13
-sun 68.9% jav→67, ind→57, unk→32, mlg→18, tgl→15, tet→14, som→12
-ita 69.0% cat→35, unk→32, mlt→20, smo→16, fra→13, som→10, tet→9
-rus 69.0% udm→51, bul→46, srp→43, che→42, chv→34, ukr→28, tgk→25
-nob 69.1% dan→96, unk→70, swe→30, fry→20, afr→16, isl→15, ita→13
-dan 69.7% nob→117, unk→53, swe→31, fry→25, afr→21, isl→11, ita→10
-nld 71.1% afr→119, fry→108, unk→23, eng→22, deu→14, yor→12, kin→11
-spa 71.6% cat→86, unk→34, por→28, tet→27, ita→22, yor→14, eus→13
-tur 71.7% aze→69, tuk→66, unk→45, sna→31, ita→11, uzb→6, sqi→5
-fas 72.1% pus→92, snd→91, azb→51, urd→13, ckb→10, unk→4, som→2
-ind 72.2% sun→73, jav→54, tet→39, unk→25, trv→12, smo→10, tgl→8
-azb 73.0% fas→228, pus→44, urd→39, snd→19, ckb→15, unk→3
-nep 73.3% hin→270, mar→85, unk→9
-swe 73.9% unk→70, nob→48, dan→28, fin→20, som→10, ita→10, sna→10
-afr 74.3% kin→46, nld→25, unk→24, fry→18, eng→17, som→8, lav→4
-cat 74.5% unk→32, ita→21, spa→20, por→12, fra→11, lav→10, mlg→10
-hin 74.6% mar→62, nep→55, unk→3, spa→1
-fra 74.8% unk→69, cat→42, smo→21, yor→19, mlg→19, eng→15, dag→14
-deu 76.4% unk→61, afr→60, kin→16, ita→13, lav→13, fry→12, gle→10
-hrv 76.6% slv→127, unk→22, yor→20, slk→14, eus→8, ces→8, ita→6
-yor 77.8% dag→11, unk→8, eng→5, aka→5, smo→3, tet→3, som→2
-slv 78.1% hrv→59, yor→32, slk→12, ces→10, unk→10, ita→5, sqi→3
-mar 79.1% hin→205, nep→43
-bul 79.5% mkd→74, srp→28, rus→21, tgk→19, ukr→9, sah→8, chv→6
-por 79.6% unk→43, tet→37, cat→21, ita→19, lit→18, eng→16, spa→16
-aka 80.0% unk→55, dag→20, ibo→19, eng→11, lug→9, som→7, smo→6
-tso 80.1% sna→23, nya→23, xho→16, sun→15, unk→15, lug→15, yor→14
-sna 80.2% nya→28, unk→20, xho→18, lug→12, smo→8, swh→6, tso→4
-uzb 80.3% unk→39, aze→30, tuk→28, eus→13, yor→12, tur→11, som→10
-mkd 80.5% bul→65, srp→57, tgk→12, che→9, rus→8, udm→6, bak→5
-mlt 81.5% unk→64, ita→35, som→17, dag→9, yor→8, ron→8, eng→8
-swh 81.9% sna→19, dag→15, yor→13, som→12, unk→12, nya→12, smo→11
-lug 81.9% xho→24, kin→21, unk→12, swh→8, sna→8, nya→8, som→6
-tuk 82.2% tur→40, unk→37, aze→9, ind→5, yor→4, pol→3, ita→3
-eus 82.5% unk→48, tet→23, ita→15, afr→8, yor→6, slv→6, pol→6
-ces 82.9% slk→101, slv→18, unk→17, yor→14, pol→8, hrv→8, est→4
-lus 82.9% unk→50, tso→36, eng→32, yor→9, aka→8, xho→8, dag→7
-tat 82.9% bak→66, kir→30, rus→21, sah→19, kaz→15, udm→15, unk→14
-slk 83.2% ces→49, slv→17, yor→12, unk→9, hrv→8, mlg→4, sqi→3
-srp 83.4% mkd→95, bul→23, rus→8, tgk→6, che→4, kir→3, ukr→3
-kin 83.5% xho→23, swh→12, yor→12, sna→12, unk→12, lug→12, zul→7
-est 83.6% unk→39, fin→19, som→8, smo→8, lit→7, eng→6, tet→6
-tsn 84.2% smo→34, unk→20, tet→16, lug→14, swh→13, ita→12, tgl→12
-kir 84.6% kaz→27, rus→27, sah→21, tat→16, tgk→16, mon→15, unk→14
-fin 84.9% est→55, unk→26, ita→16, som→11, smo→8, deu→5, lit→4
-snd 85.2% pus→10, ara→6, urd→6, azb→5, fas→4, unk→2, heb→1
-nya 85.2% unk→17, lug→15, xho→14, swh→13, sna→11, tso→9, smo→6
-som 85.2% unk→9, orm→7, est→3, smo→3, sna→3, xho→3, dag→2
-pus 85.3% azb→38, fas→33, unk→17, snd→13, urd→8, ckb→5, som→2
-aze 85.4% tur→43, unk→14, tuk→7, uzb→6, eus→4, swh→3, ita→3
-smo 85.5% ibo→11, unk→10, gle→3, tet→3, mlg→3, yor→3, sqi→2
-urd 85.8% snd→135, fas→17, azb→16, unk→2, ckb→2, som→1, spa→1
-tgl 86.2% hil→79, trv→8, sun→6, ibo→5, lus→4, som→4, unk→4
-ron 86.4% unk→22, ita→15, cat→13, smo→11, fra→9, tet→7, tur→5
-bak 86.4% tat→37, kaz→24, rus→12, bul→12, unk→10, kir→8, sah→6
-lit 87.3% unk→36, lav→19, smo→9, ita→8, slv→7, est→5, cat→4
-ukr 87.3% srp→35, rus→25, bul→23, bel→16, tgk→15, mkd→12, chv→8
-sqi 88.2% unk→18, slv→9, ita→7, som→5, yor→5, hrv→4, swh→3
-lav 88.6% unk→20, lit→16, hrv→5, est→4, cym→4, fin→3, slv→3
-hun 89.3% unk→35, slk→7, ita→6, eus→5, cat→4, eng→4, tet→4
-ibo 89.4% yor→9, unk→8, smo→4, nya→4, swh→3, hil→3, som→3
-hau 89.4% swh→13, unk→12, som→9, trv→9, dag→8, sna→7, ita→6
-mlg 90.0% smo→7, tet→5, unk→4, spa→2, hrv→2, tso→2, aka→2
-ewe 90.2% unk→35, aka→10, yor→8, ibo→6, eus→4, dag→4, smo→4
-kaz 90.2% kir→20, unk→18, bak→13, rus→9, tgk→9, bul→8, ukr→6
-pol 90.2% unk→18, ces→9, hrv→9, slk→9, slv→8, ita→6, xho→5
-tgk 92.4% che→7, bul→7, mon→3, unk→3, udm→3, mkd→2, srp→2
-isl 92.7% unk→16, eng→6, nob→4, ita→4, hun→3, tso→3, lit→3
-gle 92.7% unk→8, eng→5, lus→4, spa→2, sun→2, ron→2, smo→2
-ara 93.4% snd→39, urd→27, pus→26, fas→14, azb→4, unk→2, ckb→2
-amh 93.5% tir→16, unk→9, orm→3, eng→3, lus→2, tet→1, spa→1
-lao 93.9% unk→99, ita→3, spa→2, sna→2, xho→2, eng→2, gle→1
-tir 94.0% amh→96, lav→1, unk→1
-bel 94.2% rus→14, ukr→11, mkd→5, sah→5, udm→5, che→4, mon→3
-cym 94.5% gle→8, unk→6, pol→3, ita→3, lav→2, spa→1, mlg→1
-mon 94.8% sah→19, che→8, tgk→6, kir→5, rus→4, oss→4, udm→3
-asm 94.9% ben→52, unk→2
-ben 95.0% asm→47, unk→1
-
-CharSoup top confusions (languages with F1 < 95%, full):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-kur 0.0% tuk→342, yor→77, tur→73, hau→53, trv→46, kin→44, sna→33
-tuk 85.2% (no misses recorded)
-azb 91.8% fas→148, pus→3
-fas 92.9% pus→2, eng→1
-zul 93.4% xho→34, som→1, sna→1, yor→1, lug→1
-xho 93.5% zul→64, kin→1, sna→1, yor→1
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/flores-STANDARD.log b/docs/modules/ROOT/pages/advanced/lang-detection/flores-STANDARD.log
deleted file mode 100644
index ec18506367..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/flores-STANDARD.log
+++ /dev/null
@@ -1,488 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-=== Language Detection Comparison Report ===
-
-Test sentences: 203,381
-CharSoup ∩ OpenNLP: 105 languages, 104,684 sentences
-CharSoup ∩ Lingua: 71 languages, 70,784 sentences
-CharSoup ∩ Optimaize: 63 languages, 61,811 sentences
-
-Model heap (approx):
- CharSoup: ~0.0 MB
- OpenNLP: ~74.2 MB
- Lingua: ~0.1 MB (low accuracy mode)
- Optimaize: ~94.8 MB
-
-Coverage-adjusted accuracy — each detector scored on its own supported languages only
- (test sentences whose true language is not in a detector's covered set are skipped)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─ CS(ms) ON(ms) Li(ms) Opt(ms) CS sent/s
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------------------------------------------------------
-@20 78.71% 75.20% 74.87% 72.85% 76.35% 75.88% 84.87% 84.38% 1,714 1,271 12,700 659 75,034
-@50 93.73% 93.34% 86.09% 85.43% 90.99% 90.63% 94.44% 94.29% 1,704 1,446 28,231 3,489 75,474
-@100 96.67% 96.69% 90.25% 90.12% 95.43% 95.33% 96.51% 96.50% 1,947 2,400 40,830 3,668 66,054
-@150 97.08% 97.14% 90.98% 90.97% 96.15% 96.11% 96.72% 96.76% 2,083 3,141 52,379 4,046 61,742
-@200 97.14% 97.21% 91.11% 91.11% 96.23% 96.20% 96.75% 96.79% 2,334 3,907 53,287 4,046 55,102
-@500 97.14% 97.21% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 2,324 4,097 55,257 4,104 55,339
-full 97.14% 97.21% 91.12% 91.12% 96.25% 96.22% 96.76% 96.81% 2,297 4,065 54,904 4,257 55,990
-
-Breadth-weighted accuracy — all 203 FLORES languages, unsupported languages score 0
- (penalises limited coverage; use this to compare total useful output across all inputs)
- ─ CharSoup ─ ─ OpenNLP ─ ── Lingua ── ─ Optimaize ─
-Length mF1 acc mF1 acc mF1 acc mF1 acc
-----------------------------------------------------------------------
-@20 49.63% 47.56% 42.05% 40.71% 27.46% 27.53% 26.76% 26.89%
-@50 59.10% 59.02% 48.35% 47.74% 32.72% 32.88% 29.77% 30.04%
-@100 60.95% 61.14% 50.68% 50.36% 34.32% 34.58% 30.43% 30.75%
-@150 61.21% 61.43% 51.09% 50.84% 34.58% 34.86% 30.49% 30.83%
-@200 61.25% 61.47% 51.16% 50.91% 34.61% 34.90% 30.50% 30.84%
-@500 61.25% 61.47% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-full 61.25% 61.47% 51.17% 50.92% 34.61% 34.90% 30.51% 30.84%
-
-Strict accuracy — CharSoup ∩ OpenNLP (105 languages, 104,684 sentences)
- ── CharSoup ── ── OpenNLP ── CS(ms) OpenNLP(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 80.17% 75.59% 76.69% 74.07% 675 428 155,087
-@50 94.95% 94.11% 87.27% 86.37% 805 659 130,042
-@100 97.68% 97.42% 91.05% 90.79% 990 1,349 105,741
-@150 98.05% 97.87% 91.68% 91.55% 1,211 1,839 86,444
-@200 98.11% 97.94% 91.77% 91.65% 1,209 2,078 86,587
-@500 98.12% 97.95% 91.78% 91.67% 1,282 2,139 81,657
-full 98.12% 97.95% 91.78% 91.67% 1,210 2,118 86,516
-
-Strict accuracy — CharSoup ∩ Lingua (71 languages, 70,784 sentences)
- ── CharSoup ── ── Lingua ── CS(ms) Lingua(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 81.50% 75.81% 77.78% 76.80% 425 3,670 166,551
-@50 96.03% 94.91% 92.24% 91.51% 525 7,535 134,827
-@100 98.58% 98.23% 96.58% 96.20% 618 12,615 114,537
-@150 98.95% 98.70% 97.27% 96.96% 781 16,418 90,633
-@200 99.01% 98.77% 97.37% 97.07% 781 17,072 90,633
-@500 99.01% 98.77% 97.38% 97.09% 804 17,688 88,040
-full 99.01% 98.77% 97.38% 97.09% 808 17,795 87,604
-
-Strict accuracy — CharSoup ∩ Optimaize (63 languages, 61,811 sentences)
- ── CharSoup ── ── Optimaize ── CS(ms) Optimaize(ms) CS sent/s
-Length mF1 acc mF1 acc
-------------------------------------------------------------------------
-@20 82.07% 76.07% 86.30% 85.54% 371 109 166,606
-@50 96.50% 95.20% 95.40% 95.14% 469 571 131,793
-@100 98.79% 98.37% 97.23% 97.17% 537 536 115,104
-@150 99.12% 98.82% 97.33% 97.34% 662 605 93,370
-@200 99.16% 98.88% 97.34% 97.35% 699 629 88,428
-@500 99.17% 98.88% 97.34% 97.35% 670 641 92,255
-full 99.17% 98.88% 97.34% 97.35% 707 629 87,427
-
-CharSoup timing (wall-clock, full pipeline including script gate + group collapse):
-Length Wall(ms) Sent/sec
---------------------------------
-@20 1,714 75,034
-@50 1,704 75,474
-@100 1,947 66,054
-@150 2,083 61,742
-@200 2,334 55,102
-@500 2,324 55,339
-full 2,297 55,990
-
-Per-language CharSoup F1 by length:
-Language @20 @50 @100 @150 @200 @500 full
-----------------------------------------------------------------------
-ace 76.26% 93.06% 97.64% 98.22% 98.37% 98.37% 98.37%
-afr 73.31% 96.59% 99.50% 99.75% 99.70% 99.70% 99.70%
-aka 81.19% 95.26% 98.45% 98.90% 98.86% 98.86% 98.86%
-amh 89.37% 98.38% 99.45% 99.55% 99.55% 99.55% 99.55%
-ara 91.12% 99.85% 100.00% 100.00% 100.00% 100.00% 100.00%
-asm 94.67% 99.65% 99.95% 99.95% 99.95% 99.95% 99.95%
-azb 73.62% 86.34% 91.23% 92.04% 92.22% 92.16% 92.16%
-aze 87.15% 98.40% 99.80% 99.85% 99.85% 99.85% 99.85%
-bak 86.79% 98.20% 99.85% 99.95% 99.95% 99.95% 99.95%
-ban 57.66% 86.45% 95.29% 96.55% 96.66% 96.71% 96.71%
-bel 83.93% 99.65% 99.95% 100.00% 100.00% 100.00% 100.00%
-ben 94.94% 99.65% 99.95% 99.95% 99.95% 99.95% 99.95%
-bjn 52.30% 83.60% 93.17% 95.49% 95.63% 95.63% 95.63%
-bul 78.96% 97.42% 99.55% 99.65% 99.65% 99.65% 99.65%
-cat 72.43% 95.90% 99.30% 99.80% 99.80% 99.80% 99.80%
-ceb 72.43% 92.91% 98.69% 99.30% 99.35% 99.35% 99.35%
-ces 79.96% 97.78% 99.85% 99.85% 99.85% 99.85% 99.85%
-ckb 94.15% 99.70% 99.95% 99.95% 99.95% 99.95% 99.95%
-cym 95.48% 99.55% 100.00% 100.00% 100.00% 100.00% 100.00%
-dan 58.57% 90.24% 96.85% 98.15% 98.20% 98.20% 98.20%
-deu 68.59% 95.30% 99.45% 99.80% 99.90% 99.90% 99.90%
-ell 98.52% 99.75% 100.00% 100.00% 100.00% 100.00% 100.00%
-eng 45.49% 84.27% 93.57% 95.23% 95.23% 95.28% 95.28%
-epo 74.57% 95.98% 99.29% 99.39% 99.39% 99.39% 99.39%
-est 78.13% 97.72% 99.65% 99.80% 99.80% 99.80% 99.80%
-eus 77.73% 97.90% 99.65% 99.95% 99.95% 99.95% 99.95%
-ewe 90.09% 98.84% 99.85% 99.95% 99.95% 99.95% 99.95%
-fao 76.64% 96.22% 99.20% 99.45% 99.45% 99.45% 99.45%
-fas 73.82% 88.46% 92.66% 93.28% 93.41% 93.37% 93.37%
-fin 71.78% 97.44% 99.45% 99.70% 99.70% 99.70% 99.70%
-fra 66.21% 96.26% 99.29% 99.70% 99.70% 99.70% 99.70%
-gla 87.01% 98.89% 100.00% 99.95% 100.00% 100.00% 100.00%
-gle 90.34% 99.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-glg 62.23% 92.79% 98.75% 99.30% 99.45% 99.50% 99.50%
-grn 83.31% 97.66% 99.80% 99.85% 99.85% 99.85% 99.85%
-guj 99.09% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-hau 84.58% 98.56% 99.90% 99.95% 99.95% 99.95% 99.95%
-heb 94.84% 99.50% 99.70% 99.80% 99.80% 99.80% 99.80%
-hin 65.69% 83.63% 93.92% 95.88% 96.30% 96.30% 96.30%
-hrv 74.73% 96.19% 99.40% 99.70% 99.70% 99.70% 99.70%
-hun 87.64% 98.79% 99.95% 100.00% 100.00% 100.00% 100.00%
-hye 99.10% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-ibo 87.07% 98.85% 99.95% 99.95% 99.95% 99.95% 99.95%
-ilo 83.67% 97.67% 99.55% 99.70% 99.75% 99.75% 99.75%
-ind 35.66% 65.80% 77.27% 81.94% 82.67% 82.76% 82.76%
-isl 82.38% 97.46% 99.45% 99.60% 99.60% 99.60% 99.60%
-ita 62.87% 92.38% 98.27% 98.73% 98.83% 98.83% 98.83%
-jav 59.46% 88.24% 96.32% 97.43% 97.53% 97.63% 97.63%
-jpn 95.41% 98.54% 98.59% 98.59% 98.59% 98.59% 98.59%
-kab 84.64% 98.68% 99.70% 99.85% 99.85% 99.85% 99.85%
-kan 97.90% 99.80% 100.00% 100.00% 100.00% 100.00% 100.00%
-kat 99.09% 99.75% 99.90% 99.90% 99.90% 99.90% 99.90%
-kaz 89.10% 99.30% 100.00% 100.00% 100.00% 100.00% 100.00%
-khm 98.11% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-kin 82.79% 98.14% 99.85% 99.90% 99.90% 99.90% 99.90%
-kir 86.28% 98.69% 99.95% 99.95% 99.95% 99.95% 99.95%
-kor 99.30% 99.95% 99.95% 99.95% 99.95% 99.95% 99.95%
-kur 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
-lao 95.49% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-lav 85.65% 99.45% 100.00% 100.00% 100.00% 100.00% 100.00%
-lim 71.32% 94.00% 98.14% 98.59% 98.64% 98.64% 98.64%
-lit 85.68% 98.58% 99.80% 99.80% 99.80% 99.80% 99.80%
-ltz 77.82% 96.93% 99.65% 99.85% 99.85% 99.85% 99.85%
-lug 84.79% 98.07% 99.45% 99.50% 99.50% 99.50% 99.50%
-lus 79.50% 97.34% 99.60% 99.90% 99.90% 99.90% 99.90%
-mal 99.24% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-mar 73.90% 92.48% 97.95% 98.55% 98.65% 98.65% 98.65%
-min 61.76% 91.73% 96.71% 98.13% 98.28% 98.28% 98.28%
-mkd 80.70% 96.67% 99.50% 99.85% 99.85% 99.85% 99.85%
-mlg 91.10% 98.90% 99.85% 99.95% 99.95% 99.95% 99.95%
-mlt 75.97% 98.54% 99.65% 99.50% 99.60% 99.60% 99.60%
-mon 93.11% 99.60% 100.00% 100.00% 100.00% 100.00% 100.00%
-msa 58.11% 76.49% 85.64% 88.13% 88.62% 88.72% 88.72%
-mya 98.47% 99.90% 100.00% 100.00% 100.00% 100.00% 100.00%
-nep 69.47% 91.07% 97.64% 98.60% 98.70% 98.70% 98.70%
-nld 47.40% 82.02% 93.04% 95.43% 96.02% 95.97% 95.97%
-nno 61.95% 87.55% 95.93% 96.76% 97.13% 97.13% 97.13%
-nob 50.79% 82.97% 93.86% 95.59% 95.98% 95.98% 95.98%
-nso 63.19% 89.17% 95.23% 96.53% 97.01% 97.01% 97.01%
-nya 67.32% 84.17% 91.18% 91.97% 92.25% 92.29% 92.29%
-ori 98.58% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-pan 99.14% 99.95% 100.00% 100.00% 100.00% 100.00% 100.00%
-pap 70.81% 95.68% 99.35% 99.45% 99.50% 99.50% 99.50%
-pol 75.78% 92.66% 97.45% 98.22% 98.22% 98.17% 98.17%
-por 66.83% 94.80% 99.09% 99.50% 99.60% 99.65% 99.65%
-pus 86.74% 97.41% 99.75% 99.95% 99.95% 99.95% 99.95%
-ron 83.85% 98.79% 100.00% 100.00% 100.00% 100.00% 100.00%
-rus 74.11% 96.80% 99.70% 99.70% 99.70% 99.70% 99.70%
-san 62.58% 83.35% 93.60% 95.33% 95.71% 95.71% 95.71%
-sat 99.95% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-sin 99.45% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-slk 80.49% 97.90% 99.80% 99.80% 99.80% 99.80% 99.80%
-slv 77.68% 95.81% 99.20% 99.45% 99.45% 99.45% 99.45%
-smo 86.05% 98.41% 99.55% 99.85% 99.85% 99.85% 99.85%
-sna 81.06% 97.57% 99.40% 99.55% 99.55% 99.55% 99.55%
-snd 89.84% 99.20% 100.00% 100.00% 100.00% 100.00% 100.00%
-som 88.51% 99.10% 99.85% 99.80% 99.80% 99.80% 99.80%
-spa 64.65% 93.45% 99.30% 99.65% 99.70% 99.70% 99.70%
-sqi 87.86% 99.20% 99.80% 99.90% 99.90% 99.90% 99.90%
-srp 83.69% 97.76% 99.65% 99.90% 99.90% 99.90% 99.90%
-sun 61.45% 87.86% 95.87% 97.38% 97.53% 97.58% 97.58%
-swe 60.34% 92.72% 98.47% 98.78% 98.78% 98.78% 98.78%
-swh 76.96% 96.34% 99.55% 99.60% 99.60% 99.60% 99.60%
-szl 74.87% 92.81% 97.53% 98.42% 98.42% 98.37% 98.37%
-tam 98.88% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
-tat 82.31% 97.76% 99.80% 99.90% 99.90% 99.90% 99.90%
-tel 98.06% 99.75% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgk 91.41% 99.40% 100.00% 100.00% 100.00% 100.00% 100.00%
-tgl 73.78% 95.12% 99.55% 99.80% 99.80% 99.80% 99.80%
-tha 98.32% 99.85% 100.00% 100.00% 100.00% 100.00% 100.00%
-tir 91.88% 98.95% 99.65% 99.70% 99.70% 99.70% 99.70%
-tsn 70.85% 90.28% 94.58% 95.77% 96.09% 96.14% 96.14%
-tso 77.98% 97.18% 99.40% 99.65% 99.65% 99.65% 99.65%
-tuk 83.96% 97.93% 99.70% 99.85% 99.85% 99.85% 99.85%
-tum 61.16% 81.75% 89.89% 91.00% 91.37% 91.43% 91.43%
-tur 73.64% 95.56% 99.30% 99.35% 99.35% 99.35% 99.35%
-uig 98.49% 99.95% 100.00% 100.00% 100.00% 100.00% 100.00%
-ukr 81.96% 98.79% 99.80% 99.80% 99.80% 99.80% 99.80%
-urd 81.32% 93.69% 96.82% 97.76% 97.86% 97.91% 97.91%
-uzb 75.21% 97.66% 99.80% 99.80% 99.80% 99.80% 99.80%
-vie 94.90% 99.75% 100.00% 100.00% 100.00% 100.00% 100.00%
-war 73.77% 94.28% 98.11% 98.88% 98.88% 98.88% 98.88%
-xho 66.18% 83.84% 93.15% 95.03% 95.46% 95.42% 95.42%
-ydd 95.71% 99.55% 99.70% 99.80% 99.80% 99.80% 99.80%
-yor 85.85% 96.95% 99.75% 99.80% 99.80% 99.80% 99.80%
-yue 4.55% 1.97% 1.97% 1.97% 1.97% 1.97% 1.97%
-zho 76.79% 79.53% 79.85% 79.85% 79.85% 79.85% 79.85%
-zul 46.36% 80.37% 92.68% 94.62% 95.09% 95.04% 95.04%
-
-Per-language macro F1 (full):
-Language CharSoup OpenNLP Lingua Optimaize
-----------------------------------------------------------
-ace 98.37% N/A N/A N/A
-afr 99.70% 96.34% 96.40% 98.61%
-aka 98.86% N/A N/A N/A
-amh 99.55% 99.95% N/A N/A
-ara 100.00% 100.00% 100.00% 100.00%
-asm 99.95% 99.90% N/A N/A
-azb 92.16% N/A N/A N/A
-aze 99.85% 99.25% 98.99% N/A
-bak 99.95% 97.85% N/A N/A
-ban 96.71% 43.66% N/A N/A
-bel 100.00% 100.00% 100.00% 100.00%
-ben 99.95% 99.85% 100.00% 100.00%
-bjn 95.63% N/A N/A N/A
-bul 99.65% 98.33% 98.11% 99.14%
-cat 99.80% 98.03% 98.43% 86.74%
-ceb 99.35% 32.41% N/A N/A
-ces 99.85% 99.50% 98.79% 99.90%
-ckb 99.95% 0.00% N/A N/A
-cym 100.00% 99.95% 98.66% 99.85%
-dan 98.20% 95.45% 95.01% 98.06%
-deu 99.90% 99.30% 99.65% 99.40%
-ell 100.00% 99.95% 100.00% 100.00%
-eng 95.28% 97.22% 97.85% 98.13%
-epo 99.39% 98.75% 98.34% N/A
-est 99.80% 78.32% 99.50% 99.80%
-eus 99.95% 98.90% 99.15% 99.85%
-ewe 99.95% N/A N/A N/A
-fao 99.45% 97.72% N/A N/A
-fas 93.37% 0.59% 99.30% 99.95%
-fin 99.70% 99.39% 99.60% 99.65%
-fra 99.70% 99.25% 99.40% 99.55%
-gla 100.00% 99.55% N/A N/A
-gle 100.00% 99.50% 99.65% 99.95%
-glg 99.50% 95.11% N/A 97.22%
-grn 99.85% N/A N/A N/A
-guj 100.00% 100.00% 100.00% 100.00%
-hau 99.95% 97.59% N/A N/A
-heb 99.80% 99.85% 100.00% 99.95%
-hin 96.30% 89.24% 88.27% 99.90%
-hrv 99.70% 66.67% 68.24% 98.65%
-hun 100.00% 99.75% 99.80% 100.00%
-hye 100.00% 100.00% 100.00% N/A
-ibo 99.95% 99.55% N/A N/A
-ilo 99.75% N/A N/A N/A
-ind 82.76% 36.55% 78.31% 69.10%
-isl 99.60% 97.91% 99.70% 100.00%
-ita 98.83% 97.93% 98.30% 99.25%
-jav 97.63% 73.06% N/A N/A
-jpn 98.59% 99.65% 100.00% 68.12%
-kab 99.85% N/A N/A N/A
-kan 100.00% 100.00% N/A 100.00%
-kat 99.90% 100.00% 100.00% N/A
-kaz 100.00% 99.40% 97.61% N/A
-khm 100.00% 99.95% N/A 100.00%
-kin 99.90% 99.04% N/A N/A
-kir 99.95% 98.78% N/A N/A
-kor 99.95% 99.55% 100.00% 99.80%
-kur 0.00% 96.79% N/A N/A
-lao 100.00% 99.95% N/A N/A
-lav 100.00% 67.86% 99.30% 100.00%
-lim 98.64% 86.96% N/A N/A
-lit 99.80% 99.24% 99.50% 99.90%
-ltz 99.85% 98.99% N/A N/A
-lug 99.50% 97.64% 98.84% N/A
-lus 99.90% N/A N/A N/A
-mal 100.00% 100.00% N/A 100.00%
-mar 98.65% 97.97% 89.67% 100.00%
-min 98.28% 80.27% N/A N/A
-mkd 99.85% 97.59% 97.23% 96.98%
-mlg 99.95% 98.13% N/A N/A
-mlt 99.60% 99.34% N/A 99.90%
-mon 100.00% 99.90% 99.15% N/A
-msa 88.72% 67.94% 80.78% 40.12%
-mya 100.00% 100.00% N/A N/A
-nep 98.70% 97.56% N/A 99.90%
-nld 95.97% 90.66% 96.43% 98.53%
-nno 97.13% 88.95% 89.86% N/A
-nob 95.98% 87.81% 87.90% N/A
-nso 97.01% 95.91% N/A N/A
-nya 92.29% N/A N/A N/A
-ori 100.00% 100.00% N/A N/A
-pan 100.00% 99.95% 100.00% 100.00%
-pap 99.50% N/A N/A N/A
-pol 98.17% 99.70% 99.65% 99.95%
-por 99.65% 98.03% 98.40% 98.19%
-pus 99.95% 95.25% N/A N/A
-ron 100.00% 98.95% 97.34% 100.00%
-rus 99.70% 98.22% 98.51% 99.65%
-san 95.71% 84.87% N/A N/A
-sat 100.00% N/A N/A N/A
-sin 100.00% 100.00% N/A N/A
-slk 99.80% 99.14% 98.55% 99.70%
-slv 99.45% 97.72% 98.03% 98.60%
-smo 99.85% N/A N/A N/A
-sna 99.55% N/A 99.00% N/A
-snd 100.00% 99.95% N/A N/A
-som 99.80% 99.60% 99.80% 99.95%
-spa 99.70% 84.02% 98.05% 92.82%
-sqi 99.90% 99.70% 99.70% 100.00%
-srp 99.90% 99.14% 97.61% 97.28%
-sun 97.58% N/A N/A N/A
-swe 98.78% 98.42% 98.45% 99.50%
-swh 99.60% N/A N/A N/A
-szl 98.37% N/A N/A N/A
-tam 100.00% 100.00% 100.00% 100.00%
-tat 99.90% 97.72% N/A N/A
-tel 100.00% 99.55% 99.95% 100.00%
-tgk 100.00% 100.00% N/A N/A
-tgl 99.80% 54.09% 96.87% 99.65%
-tha 100.00% 99.55% 97.90% 100.00%
-tir 99.70% N/A N/A N/A
-tsn 96.14% 95.18% 92.46% N/A
-tso 99.65% N/A 98.90% N/A
-tuk 99.85% 99.75% N/A N/A
-tum 91.43% N/A N/A N/A
-tur 99.35% 98.53% 98.85% 99.95%
-uig 100.00% 91.88% N/A N/A
-ukr 99.80% 99.60% 97.86% 99.85%
-urd 97.91% 96.75% 99.30% 100.00%
-uzb 99.80% 98.21% N/A N/A
-vie 100.00% 99.90% 99.05% 100.00%
-war 98.88% 11.58% N/A N/A
-xho 95.42% 84.42% 90.94% N/A
-ydd 99.80% N/A N/A N/A
-yor 99.80% 98.58% 97.38% N/A
-yue 1.97% N/A N/A N/A
-zho 79.85% N/A 100.00% 87.76%
-zul 95.04% 81.80% 91.63% N/A
-
-CharSoup top confusions (languages with F1 < 95%, @20):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-kur 0.0% diq→546, unk→166, wln→68, cor→22, ile→10, sun→10, vol→9
-yue 4.5% zho→920, unk→26, jpn→11, ita→3, mon→2, lav→1, cdo-x-rom→1
-ind 35.7% msa→230, unk→155, bjn→46, pam→28, ace→26, ban→24, jav→24
-eng 45.5% unk→416, aka→21, ita→15, ltz→14, dag→10, pam→8, lat→7
-zul 46.4% xho→479, unk→55, yor→33, lug→19, tso→10, kin→9, swh→8
-nld 47.4% unk→218, vls→118, lim→52, fry→29, afr→26, nds→24, pap→21
-nob 50.8% unk→184, nno→111, dan→81, swe→18, wln→13, bcl→13, bre→12
-bjn 52.3% unk→136, msa→55, sun→34, pam→25, szy→25, jav→22, ind→20
-ban 57.7% unk→143, jav→63, bjn→47, pam→45, msa→33, ind→22, sun→16
-msa 58.1% ind→91, unk→85, bjn→27, ace→26, jav→16, pam→13, ban→12
-dan 58.6% unk→213, nob→70, nno→36, swe→17, bcl→15, fra→10, fao→10
-jav 59.5% unk→136, pam→41, bjn→30, ban→28, sun→23, msa→23, sqi→17
-swe 60.3% unk→206, nno→48, dan→25, fin→18, nob→18, vep→17, nds→17
-tum 61.2% nya→316, swh→58, unk→39, xho→13, kin→10, tso→9, sna→9
-sun 61.4% unk→118, bjn→40, jav→38, msa→28, pam→26, ban→22, szy→15
-min 61.8% unk→126, bjn→87, msa→32, ind→27, jav→21, bcl→19, pam→18
-nno 61.9% unk→144, nob→84, swe→25, dan→20, bcl→12, bre→9, fao→9
-glg 62.2% unk→162, spa→63, por→44, arg→32, fra→20, ita→20, eng→17
-san 62.6% hin→327, mar→90, nep→57, gom→16, unk→7, ltz→1, fra→1
-ita 62.9% unk→170, cos→16, bre→15, ina→14, fra→13, lfn→12, lat→11
-nso 63.2% tsn→200, unk→98, ibo→25, smo→18, tso→16, ewe→15, swh→12
-spa 64.7% unk→146, lfn→36, cat→21, fra→15, ita→15, arg→15, ext→14
-hin 65.7% mar→52, nep→44, gom→15, san→12, unk→1
-xho 66.2% unk→57, zul→25, tso→13, nya→11, yor→9, kin→8, sna→6
-fra 66.2% unk→173, wln→48, bre→15, ina→10, roh→9, smo→9, spa→7
-por 66.8% unk→185, glg→68, spa→19, mwl→15, arg→14, ita→13, cat→12
-nya 67.3% unk→65, tum→51, swh→33, xho→14, sna→9, tso→6, ibo→6
-deu 68.6% unk→201, bar→27, lat→27, afr→18, ita→17, gsw→16, nds→15
-nep 69.5% hin→266, mar→74, san→42, gom→7, unk→5
-pap 70.8% unk→152, ita→37, ido→15, spa→12, wln→11, lav→7, lfn→7
-tsn 70.8% unk→81, nso→30, smo→26, swh→17, hau→16, ewe→14, ibo→14
-lim 71.3% unk→143, vls→34, nds→15, nld→11, stq→10, wln→10, ron→10
-fin 71.8% unk→115, olo→39, vep→37, smn→35, vro→22, est→20, ita→12
-ceb 72.4% tgl→82, hil→80, bcl→53, unk→32, pam→29, ilo→14, war→9
-cat 72.4% unk→148, bre→23, spa→23, fra→20, lfn→19, ita→13, wln→11
-afr 73.3% unk→124, kin→35, pap→22, vls→20, eng→17, nld→16, nds→9
-azb 73.6% fas→210, urd→34, pus→31, ckb→21, pnb→7, snd→6, ara→1
-tur 73.6% unk→92, aze→50, diq→20, tuk→19, sna→12, tyv→8, hau→6
-war 73.8% bcl→71, unk→59, hil→53, bre→34, ceb→30, ilo→17, pam→8
-tgl 73.8% ceb→75, pam→66, bcl→49, hil→29, unk→25, ilo→13, lus→8
-fas 73.8% azb→75, pus→60, snd→30, urd→26, ckb→25, pnb→7, unk→5
-mar 73.9% hin→192, nep→32, san→31, gom→29, unk→1
-rus 74.1% udm→36, che→33, bul→27, myv→22, chv→20, ukr→20, mkd→19
-epo 74.6% unk→139, som→25, yor→18, lfn→18, ido→17, lat→16, spa→11
-hrv 74.7% slv→115, unk→89, slk→12, pol→8, yor→7, sqi→7, szl→7
-szl 74.9% pol→178, unk→73, csb→12, hak-x-rom→9, lav→6, hsb→6, dsb→5
-uzb 75.2% unk→115, kaa→44, aze→25, hau→21, bar→11, tur→9, avk→6
-pol 75.8% unk→77, szl→23, hak-x-rom→22, csb→22, slv→9, lav→8, dsb→7
-mlt 76.0% unk→164, avk→11, ita→10, som→9, fra→9, wln→9, ron→9
-ace 76.3% unk→83, sun→53, msa→18, ind→13, ban→9, jav→9, bjn→9
-fao 76.6% isl→106, unk→78, nno→56, nob→6, ita→5, eng→5, bar→4
-zho 76.8% unk→81, yue→36, jpn→28, szy→4, ita→3, hin→2, chv→2
-swh 77.0% unk→48, nya→22, sna→11, tum→10, hau→8, xho→7, dag→6
-slv 77.7% unk→57, hrv→54, yor→11, hsb→10, ces→8, slk→8, sqi→5
-eus 77.7% unk→171, bre→28, vep→12, ita→5, bar→4, lat→4, tet→4
-ltz 77.8% unk→105, fra→14, gsw→10, deu→10, nds→8, bre→7, ksh→6
-tso 78.0% unk→66, sna→21, avk→14, swh→13, nya→13, ilo→11, xho→10
-est 78.1% unk→108, vro→48, vep→43, fin→8, olo→8, sqi→5, smo→5
-bul 79.0% mkd→65, rus→24, tgk→24, myv→19, sah→16, che→13, ukr→11
-lus 79.5% unk→103, cnh→50, eng→11, tso→10, ltz→6, msa→6, yor→5
-ces 80.0% slk→83, unk→73, hsb→22, slv→17, rue→9, hrv→6, yor→5
-slk 80.5% unk→80, ces→46, slv→19, yor→11, ron→7, ceb→5, avk→5
-mkd 80.7% bul→68, srp→25, rus→18, tgk→16, che→10, myv→10, unk→8
-sna 81.1% unk→78, nya→38, xho→14, swh→9, hau→6, tum→5, tso→5
-aka 81.2% unk→107, dag→8, orm→5, ibo→5, lus→4, ita→4, hau→4
-urd 81.3% pnb→123, snd→53, fas→22, azb→16, pus→11, ckb→3, unk→2
-ukr 82.0% rus→37, bul→18, tgk→18, myv→17, chv→14, srp→13, kaz→11
-tat 82.3% bak→71, kir→31, kaz→20, unk→20, rus→19, mon→14, sah→10
-isl 82.4% unk→48, fao→47, gla→29, nno→20, eng→4, ron→3, ita→3
-kin 82.8% unk→44, xho→22, swh→17, sna→15, ilo→8, nya→8, ita→7
-grn 83.3% unk→102, spa→13, bre→12, ita→11, eus→10, cos→5, lat→5
-ilo 83.7% unk→32, pam→8, ami→7, tgl→6, szy→6, hil→5, ceb→3
-srp 83.7% mkd→98, bul→32, rus→19, ukr→17, tgk→12, che→9, myv→8
-ron 83.9% unk→82, lat→13, fra→10, lfn→10, ita→9, spa→6, cat→5
-bel 83.9% ukr→77, rus→42, myv→30, chv→13, sah→12, kaz→11, tat→11
-tuk 84.0% unk→96, tur→55, aze→9, deu→7, lav→5, fra→4, isl→3
-hau 84.6% unk→35, swh→7, ita→5, vep→4, ltz→4, avk→3, cym→3
-kab 84.6% unk→107, xho→12, ita→10, hau→8, avk→7, vep→4, ilo→4
-lug 84.8% unk→43, xho→20, nya→13, swh→12, kin→11, pam→8, aka→8
-lav 85.6% unk→37, lit→5, slv→4, sgs→3, vep→3, cos→2, diq→2
-lit 85.7% unk→87, sgs→39, lav→35, vep→12, hrv→4, vol→3, epo→3
-yor 85.9% unk→29, ilo→9, dag→5, aka→4, ibo→3, swh→2, kin→2
-smo 86.0% unk→37, lfn→8, ina→7, pap→6, ibo→5, por→3, bre→3
-kir 86.3% mon→27, kaz→26, rus→19, unk→16, ukr→11, che→9, tat→9
-pus 86.7% azb→45, ckb→24, fas→21, pnb→10, urd→9, unk→7, snd→3
-bak 86.8% tat→27, kaz→22, rus→18, unk→12, mon→8, ukr→7, che→6
-gla 87.0% gle→65, unk→33, lus→10, eng→6, hau→4, ibo→4, vep→3
-ibo 87.1% unk→25, xho→7, swh→6, kin→4, nya→4, ita→3, yor→3
-aze 87.2% unk→41, tur→38, diq→11, uzb→10, kaa→6, vep→3, ron→2
-hun 87.6% unk→80, epo→20, bre→10, vep→4, slk→4, wln→4, avk→4
-sqi 87.9% unk→49, wln→4, epo→3, ban→2, ita→2, lav→2, vep→2
-som 88.5% unk→39, orm→18, fra→3, aka→3, msa→3, ceb→2, dag→2
-kaz 89.1% bak→15, kir→15, rus→14, unk→12, tat→10, ukr→8, mon→5
-amh 89.4% tir→142, unk→14, orm→6, lav→1, hak-x-rom→1, hye→1, urd→1
-snd 89.8% pus→23, azb→21, ara→17, fas→13, urd→8, ckb→3, pnb→2
-ewe 90.1% unk→47, aka→8, avk→7, eng→4, vol→3, hak-x-rom→3, lat→3
-gle 90.3% gla→34, unk→11, lus→4, bre→4, ltz→3, mwl→2, ron→2
-mlg 91.1% cos→28, unk→23, avk→3, smo→2, pap→2, tet→2, epo→2
-ara 91.1% ckb→34, urd→29, snd→21, pus→18, azb→15, fas→14, uig→10
-tgk 91.4% myv→7, mon→6, ukr→6, ava→5, rus→5, unk→5, kir→5
-tir 91.9% amh→27, lav→1, hye→1
-mon 93.1% che→8, myv→5, rus→5, sah→5, kir→4, oss→3, tgk→3
-ckb 94.2% azb→3, fas→2, urd→1
-asm 94.7% ben→73, unk→1
-heb 94.8% ydd→72, unk→3, hye→1, avk→1, fas→1
-vie 94.9% unk→15, hak-x-rom→3, cdo-x-rom→2, ltz→2, fra→2, ita→2, lav→1
-ben 94.9% asm→30
-
-CharSoup top confusions (languages with F1 < 95%, full):
-TrueLabel F1 Top misclassifications (predicted → count)
-------------------------------------------------------------------------
-kur 0.0% diq→987, wln→3, tur→2, som→1, tso→1, eus→1, bjn→1
-yue 2.0% zho→969, jpn→4, szy→2, cdo-x-rom→2, msa→2, pol→2, dan→1
-zho 79.8% yue→7, jpn→4, unk→3, ind→2, kat→2, szy→1, spa→1
-ind 82.8% msa→183, bjn→46, sun→6, jav→5, min→4, ace→1, ido→1
-msa 88.7% ind→31, bjn→7, jav→2, min→1
-tum 91.4% nya→149
-azb 92.2% fas→137, pnb→4, urd→3, ckb→1
-nya 92.3% tum→5, swh→2, tsn→1, lug→1
-fas 93.4% pnb→3, orm→1
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/flores200-dev-eval.md b/docs/modules/ROOT/pages/advanced/lang-detection/flores200-dev-eval.md
deleted file mode 100644
index c74eb636f5..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/flores200-dev-eval.md
+++ /dev/null
@@ -1,204 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# CharSoup Language Detection — Evaluation Report
-
-**Models**: v7 standard (203 languages, 16 384 buckets, flags 0x075) + short-text v1 (122 languages, 32 768 buckets, flags 0x0a1).
-**Training corpus**: Wikipedia + MADLAD-400 (Kudugunta et al., arXiv:2309.04662) supplements.
-**Evaluation set**: [FLORES-200](https://github.com/facebookresearch/flores) dev split (203 languages × 1 001 sentences = 203 381 sentences).
-Sentence lengths: median 124 chars, 91% under 200 chars — scores flatten from @200 onward because truncating to 200 vs 500 chars affects only the remaining 9% of sentences.
-**Raw reports**: `~/datasets/compare-v7-final/flores-STANDARD.log`,
-`~/datasets/compare-v7-final/flores-SHORT_TEXT.log`,
-`~/datasets/compare-v7-final/flores-AUTOMATIC.log`
-(generated 2026-03-10 against final shipped models)
-
-**Model provenance**:
-- General: `langdetect-v7-20260306.bin` — ScriptAwareFeatureExtractor (bigrams + trigrams + suffixes + prefixes + word unigrams + char unigrams)
-- Short-text: `langdetect-short-v1-20260310.bin` — ShortTextFeatureExtractor (trigrams + word unigrams + 4-grams). Trained 2026-03-10 on wikipedia-model-v7 pool, 122 languages (excludes `ceb`; see `short-text-language-decisions.md`).
-
----
-
-## Inference pipeline (v7)
-
-The full pipeline applied to every input chunk:
-
-1. **Feature extraction** — standard model: bigrams + trigrams + suffixes + prefixes + word unigrams + char unigrams (ScriptAwareFeatureExtractor, flags 0x075);
- short-text model: bigrams + trigrams + word unigrams + 4-grams (ShortTextFeatureExtractor, flags 0x0a1)
-2. **Raw logits** — dot-product against INT8-quantized weight matrix
-3. **Script gate** — mask languages whose expected script doesn't match the dominant
- script of the input (fires when ≥85% of characters fall in one Unicode block group)
-4. **Confusable-group collapse** — for structurally confusable pairs (`ind`/`msa`,
- `yue`/`zho`, etc.), sum logits of both members and return only the higher-scoring
- one as the group winner
-5. **Rank and return** — top-k results by logit
-
-**Automatic model selection** (Strategy.AUTOMATIC): for inputs shorter than 200 characters
-or with fewer than 200 n-gram emissions (sparse inputs), the short-text model is used
-instead of the standard model. Both models are loaded at startup; switching is zero-copy.
-
----
-
-## Detectors compared
-
-| Detector | Model size | Languages | Notes |
-|----------|-----------|-----------|-------|
-| **CharSoup standard** | ~3.2 MB | **203** | Bigrams + trigrams + suffixes + prefixes + word unigrams + char unigrams, 16k buckets |
-| **CharSoup short-text** | ~3.8 MB | **122** | Bigrams + trigrams + word unigrams + 4-grams, 32k buckets |
-| **CharSoup automatic** | ~7.0 MB combined | **203** | Routes to short-text for short/sparse input |
-| **OpenNLP** | ~76 MB | 105 (of 203) | `tika-langdetect-opennlp` |
-| **Lingua** | ~0.1 MB | 71 (of 203) | `lingua-detector`, low-accuracy mode |
-| **Optimaize** | ~95 MB | 63 (of 203) | `tika-langdetect-optimaize` |
-
----
-
-## Coverage-adjusted macro-F1
-
-Each detector is scored **only on its own supported language set** — sentences whose true
-language is not in a detector's covered set are skipped entirely. This rewards accuracy
-within coverage and prevents penalising detectors for languages they don't claim to support.
-
-| Length | CS-std (203L) | CS-short (122L) | CS-auto (203L) | OpenNLP (105L) | Lingua (71L) | Optimaize (63L) |
-|--------|:---:|:---:|:---:|:---:|:---:|:---:|
-| @20 | 78.71% | **84.68%** | 65.04%† | 74.87% | 76.35% | 84.87% |
-| @50 | 93.73% | **96.06%** | 73.72%† | 86.09% | 90.99% | 94.44% |
-| @100 | 96.67% | **97.92%** | 75.32%† | 90.25% | 95.43% | 96.51% |
-| @150 | 97.08% | **98.11%** | 75.53%† | 90.98% | 96.15% | 96.72% |
-| @200 | 97.14% | **98.14%** | 79.21%† | 91.11% | 96.23% | 96.75% |
-| @500 | 97.14% | **98.14%** | 79.22%† | 91.12% | 96.25% | 96.76% |
-
-† **CS-auto caveat**: in AUTOMATIC mode the length threshold fires for nearly every FLORES sentence
-(all are ≥ 20 chars), routing all 203 languages through the 122-language short-text model.
-The 81 languages that are in the standard model but not the short-text model receive
-suboptimal predictions, which depresses the coverage-adjusted score. In real use, those
-languages do still receive a best-effort result from the short-text model; the penalty here
-is an evaluation artefact of the threshold being set relative to natural prose lengths rather
-than FLORES sentence lengths.
-
----
-
-## Breadth-weighted macro-F1
-
-All 203 FLORES languages are included in the denominator. Languages not in a detector's
-supported set contribute 0 to the macro average. This penalises limited coverage and
-rewards both accuracy and breadth.
-
-| Length | CS-std | CS-short | CS-auto | OpenNLP | Lingua | Optimaize |
-|--------|:---:|:---:|:---:|:---:|:---:|:---:|
-| @20 | **49.63%** | 43.80% | 41.01% | 42.05% | 27.46% | 26.76% |
-| @50 | **59.10%** | 49.68% | 46.49% | 48.35% | 32.72% | 29.77% |
-| @100 | **60.95%** | 50.65% | 47.49% | 50.68% | 34.32% | 30.43% |
-| @150 | **61.21%** | 50.75% | 47.63% | 51.09% | 34.58% | 30.49% |
-| @200 | **61.25%** | 50.76% | 49.95% | 51.16% | 34.61% | 30.50% |
-| @500 | **61.25%** | 50.76% | 49.95% | 51.17% | 34.61% | 30.51% |
-
-**Key takeaway**: every detector drops substantially because none covers all 203 FLORES
-evaluation languages. CS-std leads because its 203-language model covers 128 of the 203
-FLORES codes — the remaining 75 (Arabic dialect codes, regional languages, script variants)
-are not in the model and score 0. Other detectors cover even fewer FLORES languages:
-OpenNLP 105 → covers ~105 of 203 (42.05%); CS-short 122 → covers ~105 of 203 (43.80%);
-Lingua 71 → covers ~71 of 203 (27.46%); Optimaize 63 → covers ~63 of 203 (26.76%).
-
-**CS-auto caveat**: the AUTOMATIC routing threshold fires whenever `input.length() < 200`,
-routing those sentences to the 122-language short-text model. The 81 languages in the
-standard model but not the short-text model receive wrong predictions, which depresses
-the score further than a simple 0 would.
-
----
-
-## Intersection comparisons (shared supported languages only)
-
-For a fair head-to-head, both detectors are evaluated on the **intersection** of their
-supported language sets. See the raw report files for the full pairwise intersection tables.
-
-Key intersections at @20 chars (macro-F1 on shared languages):
-
-| Pair | Shared langs | CS F1 | Other F1 |
-|------|:---:|:---:|:---:|
-| CS-std ∩ OpenNLP | 105 | 80.17% | 76.69% |
-| CS-std ∩ Lingua | 71 | 81.50% | 77.78% |
-| CS-std ∩ Optimaize | 63 | 82.07% | 86.30% |
-| CS-short ∩ OpenNLP | 91 | 85.20% | 81.12% |
-| CS-short ∩ Lingua | 67 | 85.77% | 79.29% |
-| CS-short ∩ Optimaize | 59 | 86.44% | 87.83% |
-
----
-
-## Throughput
-
-Measured on FLORES-200 dev set, multi-threaded (12 cores), Apple M-series.
-All CharSoup variants run within a single JVM with both models loaded.
-(Measured 2026-03-10; strategy evals ran sequentially. Absolute numbers
-are approximate — relative ordering is the meaningful signal.)
-
-| Length | CS-std | CS-short | CS-auto | OpenNLP | Lingua | Optimaize |
-|--------|:---:|:---:|:---:|:---:|:---:|:---:|
-| @20 | 75.0k/s | 55.4k/s | 68.8k/s | 49.1k/s | 2.8k/s | 93.8k/s |
-| @50 | 75.5k/s | 51.6k/s | 55.9k/s | 31.3k/s | 1.6k/s | 7.8k/s |
-| @100 | 66.1k/s | 41.3k/s | 58.2k/s | 21.4k/s | 1.0k/s | 8.1k/s |
-| @200 | 55.1k/s | 36.1k/s | 40.6k/s | 23.6k/s | 0.7k/s | 12.3k/s |
-| @500 | 55.3k/s | 35.1k/s | 30.1k/s | 17.1k/s | 0.8k/s | 13.7k/s |
-
-**Notes:**
-- Optimaize is very fast at @20 (94k/s) because most of its 63-language model fits in
- cache at short inputs; it degrades sharply at @50+ as scoring cost scales with length.
-- CS-short is slower than CS-std despite fewer classes (122 vs 203) because it uses
- 32k buckets (vs 16k) and extracts 4-grams in addition to trigrams and word unigrams.
-- Lingua (low-accuracy mode) is 15–50× slower than CharSoup across all lengths.
-
----
-
-## Per-language highlights (FLORES @20, CS-std vs CS-short)
-
-Languages with the largest improvement from the short-text model at 20 chars:
-
-| Language | CS-std @20 | CS-short @20 | Gain |
-|----------|:---:|:---:|:---:|
-| Portuguese (`por`) | 66.8% | 79.6% | +12.8% |
-| Dutch (`nld`) | 47.4% | 71.1% | +23.7% |
-| English (`eng`) | 45.5% | 52.4% | +6.9% |
-| Spanish (`spa`) | 64.7% | 71.6% | +6.9% |
-| French (`fra`) | 66.2% | 74.8% | +8.6% |
-| German (`deu`) | 68.6% | 76.4% | +7.8% |
-| Italian (`ita`) | 62.9% | 69.0% | +6.1% |
-
-Script-distinct languages (Arabic, Thai, Japanese, Korean, etc.) already score ≥95%
-at @20 in the standard model and see little change in the short-text model.
-
-### English @20 — remaining confusions (short-text v1)
-
-English @20 F1 = 52.4%. Top confusions:
-
-| Predicted | Count | Notes |
-|-----------|:---:|-------|
-| `unk` (below confidence) | 158 | Model not confident enough to commit — flat distribution across Latin-script languages. Dual confidence thresholds (general: 0.20, short-text: 0.10) recover ~10 sentences vs uniform threshold. |
-| `dag` (Dagbani) | 33 | |
-| `ita` (Italian) | 29 | |
-| `mlg` (Malagasy) | 22 | |
-| `yor` (Yoruba) | 17 | |
-| `aze` (Azerbaijani) | 17 | |
-| `orm` (Oromo) | 17 | |
-
-The primary bottleneck is not misclassification but **low confidence on very short
-Latin-script text**. At 20 characters, English function words (`the`, `of`, `in`, `is`)
-are shared with or resemble words in many other Latin-script languages, producing
-insufficient logit margin between candidates.
-
-**Planned improvement**: stop-word anchoring — a fixed vocabulary of high-frequency
-function words per language, hashed as exact-match features. Even at 20 chars, bigrams
-like `of the` or `in the` would provide a decisive English signal, while `van de` or
-`in het` would anchor Dutch. This addresses the `unk` problem directly without
-threshold manipulation.
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/language-drop-decisions.md b/docs/modules/ROOT/pages/advanced/lang-detection/language-drop-decisions.md
deleted file mode 100644
index ad004ce317..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/language-drop-decisions.md
+++ /dev/null
@@ -1,175 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Language Drop Decisions
-
-This document records why specific languages were removed from the CharSoup
-training pool and how closely related languages are handled at inference time.
-
-**Important framing**: every decision here is a limitation of this model —
-specifically of character n-gram features — not a statement about the languages
-themselves. Languages that share short n-gram profiles are genuinely distinct
-languages whose speakers can tell them apart immediately from broader context,
-vocabulary, and grammar. A character n-gram model working on short text snippets
-simply does not have access to that evidence. Where possible we use the gating
-mechanism below to keep languages in the model rather than drop them entirely.
-
-**Current model: 203 languages** (v7, `langdetect-v7-20260306.bin`).
-**Short-text model: 122 languages** (v1, `langdetect-short-v1-20260310.bin`; see `short-text-language-decisions.md`).
-
-History: v4 → v5 dropped 9 bot-corpus languages and 3 gate-dependent languages,
-removed the length-gating mechanism, and added `yue`/`zho` confusable pair.
-v5 → v7 added MADLAD-400 supplements and switched to ScriptAwareFeatureExtractor
-(trigrams + suffixes + prefixes + word unigrams + char unigrams, 16k buckets).
-
----
-
-## Excluded languages
-
-All exclusions are enforced in `PrepareCorpus.EXCLUDED_LANGS`. The code
-comments contain per-language rationale; this table summarises.
-
-### Insufficient character-level separation in current model
-
-These languages overlap so heavily in their short character n-gram profiles
-that keeping all members produces severe mutual accuracy loss with no reliable
-gain. The limitation is the model's feature set, not the languages.
-
-| Code | Language | Reason |
-|------|----------|--------|
-| `bos` | Bosnian | Character n-gram overlap with `hrv` and `srp` is too high for reliable short-text separation. `hrv` and `srp` are retained; `bos` documents will typically match one of them. |
-| `hbs` | Serbo-Croatian | Same character n-gram overlap problem across the whole South Slavic cluster; F1 never above 0.85. 77.6% Pass-2 retention — highest mislabel rate in pool. |
-| `dzo` | Dzongkha | Tibetan script profile overlaps `bod` severely; neither can be returned reliably when both are in the model. |
-| `bod` | Tibetan | Same issue as `dzo`; both excluded rather than producing unreliable results for either. |
-| `koi` | Komi-Permyak | 22% n-gram overlap with `kpv` (Komi-Zyrian) at 500 chars — not resolvable with bigrams alone. Both are Komi varieties; `kpv` has higher F1 at every length and is retained. |
-| `msi` | Sabah Malay | Short-text n-gram profile converges with `msa`/`ind` beyond what bigrams can separate. |
-| `meo` | Meru | Same convergence profile as `msi` with Malay-family languages. |
-| `que` | Quechua | Character n-gram separation from `aym` never reaches F1=0.85 at any text length. |
-| `aym` | Aymara | Character n-gram separation from `que` never reaches F1=0.85 at any text length. |
-
-### Accuracy too low / corpus too poor to be useful
-
-Keeping these would add a label that misfires on most inputs, which is worse
-than not covering the language at all.
-
-| Code | Language | F1@500 | Reason |
-|------|----------|--------|--------|
-| `mad` | Madurese | 0.09 | Model accuracy near-random; character profile converges with Javanese/Indonesian at all lengths. |
-| `anw` | Anaang | 0.33 | Too few clean training sentences for reliable detection at any length. |
-| `glk` | Gilaki | — | Script profile converges with Persian and Mazanderani. |
-| `bgp` | Eastern Balochi | — | Short-text n-gram profile converges with Arabic/Urdu/Persian. |
-| `hif` | Fiji Hindi | 0.81 | Character-level convergence with standard Hindi at every length. |
-| `gor` | Gorontalo | 0.80 | F1@50=0.50 — near-random at 50-char samples, no recovery at longer lengths. |
-| `pcd` | Picard | 0.81 | Character n-gram profile converges with French at all lengths; no reliable recovery. |
-| `sot` | Southern Sotho | — | Including `sot` collapsed Tswana (`tsn`) from 98% to 86% — net harm to both. |
-| `crh` | Crimean Tatar | — | Latin-script orthography produces n-gram profiles that overlap heavily with Turkish; caused ~400 Turkish false positives. |
-| `zza` | Zazaki | — | Training data contaminated with Turkish; caused ~400 Turkish false positives. |
-| `knn` | Konkani | — | Devanagari script produces profiles that converge with Marathi. |
-| `mkw` | Kituba | — | Profile converges with Kongo and Lingala. |
-| `ang` | Old English | — | Character profile converges with modern English; 109 English misclassifications. |
-| `als` | Tosk Albanian | — | Including `als` collapsed standard Albanian (`sqi`) from 99% to 51.6% — net harm to both. |
-| `vec` | Venetian | — | Including `vec` collapsed Italian (`ita`) by 14.5 pp — net harm to both. |
-| `mai` | Maithili | 0.90 | F1@50=0.49. Devanagari script profiles converge with Hindi/Nepali; 73.5% Pass-2 retention indicates substantial corpus contamination. |
-
-### Training corpus unreliable
-
-The language itself is well-defined and detectable in principle, but the
-available training data does not represent it well enough to produce a
-useful model.
-
-| Code | Language | Reason |
-|------|----------|--------|
-| `sco` | Scots | The Scots Wikipedia was largely written by a non-Scots speaker modifying English articles; the training data does not reliably represent Scots. The resulting model was simultaneously the #1 false-positive source into English and the #2 recall drain for English @20 chars. |
-| `ceb` | Cebuano | Wikipedia corpus is ~80% Lsjbot-generated municipality stubs. F1@500=0.999 on Wikipedia dev vs **13.9% on FLORES-200** — the model learned the template, not the language. MADLAD-400 data (Kudugunta et al., arXiv:2309.04662) was later tried in the short-text model but achieved only 26.9% FLORES @20 across all feature configs — still below useful threshold, with heavy `tgl` (Tagalog) confusion. |
-
-### Bot-generated or templated Wikipedia corpus (dropped in v5)
-
-These languages were identified by a systematic audit: (1) computing the gap
-between Wikipedia F1@500 (in-domain) and FLORES F1 (out-of-domain) for all
-overlapping languages, then (2) manually sampling 20–30 evenly-spaced
-sentences from each language's pool file. All languages below had their
-training data found to be overwhelmingly templated bot stubs. Reinclusion
-requires a corpus of genuine prose.
-
-| Code | Language | Corpus size | Finding |
-|------|----------|-------------|---------|
-| `arz` | Egyptian Arabic | — | Sentences follow person/place birth-stub template: "من مواليد يوم [date] سنة [year] فى [place]". Wikipedia F1@500=99.9%, FLORES F1=5.3% — same bot-overfit signature as `ceb`. |
-| `bug` | Buginese | 8k sentences | ~95% French municipality stubs: "X iyanaritu séuwa komun ri déparetema Y ri Perancis." |
-| `bpy` | Bishnupriya Manipuri | 18k sentences | Dominated by Brazilian municipality, US county, and Indian/Bangladeshi location stubs. Near-zero genuine prose. |
-| `mlg` | Malagasy | 179k sentences | French commune stubs dominate: "Ny isam-ponin'i X dia Y mponina araka ny fanisana..." Model learned the template. |
-| `nan-x-rom` | Min Nan Chinese (romanized) | 455k sentences | 6/8 random samples are geographic stubs (Romanian communes, Bolivian municipalities, German/Iranian villages). |
-| `new` | Newari | 20k sentences | Indian village stubs dominate: "भारतया X राज्यया Y जनपदया Z तहसीलया छगु गां ख". |
-| `lld` | Ladin | 143k sentences | All sampled sentences are stubs — mountain ranges, municipalities, towns, incomplete game articles. |
-| `che` | Chechen | 456k sentences | 19/20 random samples are Russian, Turkish, Belarusian, and Mexican village/town stubs written in Chechen script. |
-| `nav` | Navajo | 47k sentences | ~95% species distribution stubs: "éí [animal] dah yikahjí atah yisdzoh... Ndaʼałkaahí dóó ééʼdeetįįhii éí [scientific name] deiłníigo dayózhí". |
-
-### Below F1=0.85 threshold
-
-| Code | Language | F1@500 | Reason |
-|------|----------|--------|--------|
-| `zea` | Zeelandic | 0.827 | Produces 1.4% recall loss for Dutch (`nld`) on FLORES at full sentence length. `vls` is retained at F1@500=0.925. |
-
-### Gate-dependent languages (dropped in v5)
-
-The length-gating mechanism was removed from `CharSoupLanguageDetector`
-in v5. Languages that required gating to avoid harming high-traffic parent
-languages are dropped instead. Languages with only minor short-text bleed into
-their parent (`vls`, `ext`, `rue`, `bjn`, `nap`) are **retained** — the bleed
-at short lengths is acceptable and short-text variety detection can be
-revisited in a future model.
-
-| Code | Language | Reason for dropping |
-|------|----------|---------------------|
-| `hat` | Haitian Creole | Was the #1 false-positive source into English @20 chars without gating. Corpus is mixed (filmography stubs, place stubs, genuine prose), weakening the hat signal. Dropping rather than restoring the English bleed. |
-| `nds-nl` | Low Saxon Dutch | Had the most complex dual-confusable behavior: permanently paired with `nds` (Low Saxon German) AND requiring a short-text gate against `nld` (Dutch). nds-nl text routes to `nds` via the existing confusable pair. |
-| `map-bms` | Banyumasan | 5.7% bleed into `ind` and 3.5% into `jav` at **full length** — gating only addressed short text, the full-length bleed remained. |
-
----
-
-## Paired languages (`confusables.txt`)
-
-These are distinct, well-established languages that the model supports and
-can separate at longer lengths. However, their short character n-gram profiles
-overlap enough that a short-text coin-flip between them is not useful. The
-model combines their scores and returns the higher one, reporting confidence
-in the pair rather than a noisy single-language guess.
-
-| Pair | Notes |
-|------|-------|
-| `ind` (Indonesian) / `msa` (Malay) | These are distinct national standard languages with a shared historical root and substantial written overlap. At 500 chars, ~9% of Indonesian documents score higher for Malay and vice versa — the character bigram space does not reliably separate them at any length. Both are fully supported; the model returns whichever scores higher. |
-| `nds` (Low Saxon German) | Low Saxon German. `nds-nl` (Low Saxon Dutch) was previously paired here but was dropped in v5 (see gate-dependent drops above). |
-| `xho` (Xhosa) / `zul` (Zulu) | Both Nguni Bantu languages. Character n-gram profiles overlap heavily; reliable separation requires broader lexical and morphological context. Both fully supported. |
-| `bel` (Belarusian) / `be-x-old` (Belarusian Taraškievica) | Two written standards for Belarusian. Grouped to prevent cross-script contamination filtering during training. Both fully supported. |
-| `yue` (Cantonese) / `zho` (Mandarin) | *(Added v5)* Both written in Han script. Manual sampling confirms the `yue` Wikipedia corpus is genuine Cantonese prose (distinctive particles and vocabulary), but the shared script means character n-gram profiles overlap heavily at the bigram level. FLORES F1 for `yue` is near zero because FLORES contains only Mandarin for that script — this is an evaluation artefact, not a model failure. Both fully supported; the model returns whichever scores h [...]
-
----
-
-## Reinstated languages (previously excluded from MADLAD model)
-
-The switch to Wikipedia as the primary corpus resolved several MADLAD-specific
-data quality issues. The following languages were re-evaluated and retained:
-
-| Code | Language | Previous reason for exclusion | Resolution |
-|------|----------|-------------------------------|------------|
-| `war` | Waray | MADLAD: quality flags, elevated English overlap | Wikipedia corpus clean; 99.2% FLORES recall |
-| `iku` | Inuktitut | MADLAD: corpus contaminated with English text | Wikipedia uses Canadian Syllabics script; script gate (`CANADIAN_ABORIGINAL`) ensures Latin-script input cannot trigger `iku` |
-| `shn` | Shan | MADLAD: 23% of sentences flagged as English | Wikipedia uses Myanmar script; script gate added |
-| `cor` | Cornish | MADLAD: corpus contained substantial English code-switching | Wikipedia corpus retested; retained |
-| `glv` | Manx | MADLAD: corpus contaminated with English | Wikipedia corpus retested; retained |
-| `gsw` | Alsatian | MADLAD: quality flags, German overlap | Wikipedia corpus retested; retained |
-| `haw` | Hawaiian | MADLAD: insufficient clean data | Wikipedia corpus sufficient |
-| `bjn` | Banjar | Wikipedia round 2: short-text overlap with `ind`/`msa` | Retained in v5 without gating; minor short-text bleed into `ind` accepted |
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/short-text-language-decisions.md b/docs/modules/ROOT/pages/advanced/lang-detection/short-text-language-decisions.md
deleted file mode 100644
index 8057b8b7c5..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/short-text-language-decisions.md
+++ /dev/null
@@ -1,384 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Short-Text Language Detection — Language Selection
-
-This document records the language-inclusion policy for the **short-text
-variant** of the CharSoup language detector. Where the full v7 model targets
-broad coverage (203 languages) optimised for ≥100-character inputs, this
-model targets **reliable detection at 20–50 characters**.
-
----
-
-## Design rationale
-
-At 20 characters the v7 model (203 languages, STANDARD strategy) achieves 78.71% macro-F1. The gap relative to
-longer text is almost entirely explained by two phenomena:
-
-1. **Cluster confusion.** Major European languages (English, French, German,
- Italian, Spanish, Dutch) are surrounded by many closely related regional
- varieties — Occitan, Corsican, Lombard, Walloon, Alsatian, Low Saxon, etc.
- At 20 chars the character bigram profiles of these varieties overlap heavily
- with the language. The model splits probability mass between them,
- degrading the language's score even though those varieties are the
- minority case in real documents.
-
-2. **Low-entropy inputs.** Very short snippets can be genuine function-word
- sequences, partial tokens, or punctuation runs that carry almost no
- language-specific signal regardless of which languages are in the model.
-
-The short-text model addresses (1) directly: by removing the low-traffic
-regional varieties that cause the most confusion, the included languages reclaim
-the probability mass that was being bled away. Phenomenon (2) is irreducible;
-the model returns low confidence for those inputs regardless.
-
-**Key consequence:** languages that look weak in the v7 @20 evaluation
-(English 45.5%, French 66.2%, German 68.6%, Spanish 64.7%, Italian 62.9%,
-Dutch 47.4%) are expected to recover substantially in a model trained only
-on the languages listed here, because their primary confusers are absent.
-
----
-
-## Inclusion criteria
-
-A language is included if it satisfies **any** of the following:
-
-1. **Unique or near-unique script.** Languages whose Unicode script is shared
- by few or no other included languages achieve ~100% F1 at 20 chars
- regardless of model size; there is no reason to exclude them.
-
-2. **F1@20 ≥ 75% in the full v7 model.** If the language already detects well
- at 20 chars in the presence of 203 competitors, it will only improve in a
- smaller set.
-
-3. **Major practical language.** Languages that appear frequently in documents
- Tika processes (large web/enterprise corpus) are included even if their
- current v7 @20 F1 is below 75%, with the expectation that removing their
- confusers restores performance. This applies primarily to the European
- languages and to Indonesian/Malay.
-
-A language is **excluded** if:
-
-- It is a regional variety whose character n-gram profile closely mirrors a
- another included language at short lengths, causing that language to lose
- probability mass (see cluster exclusions below).
-- It is a classical or constructed language (Sanskrit, Esperanto, Latin).
-- It has fewer than ~10,000 clean Wikipedia sentences or its corpus is known
- to be dominated by templated bot stubs (see v7 `language-drop-decisions.md`
- for the full bot-corpus audit).
-
----
-
-## Included languages (122)
-
-### Non-Latin-script languages
-
-These are reliable at any text length due to script uniqueness. All are
-included unconditionally.
-
-| Script | Code | Language | v7 F1@20 |
-|--------|------|----------|----------|
-| Arabic | `ara` | Arabic | ~100% |
-| Arabic | `azb` | South Azerbaijani | 79.2% |
-| Arabic | `ckb` | Central Kurdish (Sorani) | ~100% |
-| Arabic | `fas` | Persian | ~100% |
-| Arabic | `pus` | Pashto | ~100% |
-| Arabic | `snd` | Sindhi | 91.9% |
-| Arabic | `uig` | Uyghur | ~100% |
-| Arabic | `urd` | Urdu | 84.7% |
-| Bengali | `asm` | Assamese | 94.5% |
-| Bengali | `ben` | Bengali | 94.7% |
-| Han/CJK | `yue` | Cantonese | 5.4% (confusable group with `zho`) |
-| Han/CJK | `zho` | Chinese (Mandarin) | 77.2% |
-| Cyrillic | `bak` | Bashkir | 86.0% |
-| Cyrillic | `bel` | Belarusian | 82.4% |
-| Cyrillic | `bul` | Bulgarian | 77.8% |
-| Cyrillic | `chv` | Chuvash | — |
-| Cyrillic | `kaz` | Kazakh | 88.6% |
-| Cyrillic | `kir` | Kyrgyz | 85.3% |
-| Cyrillic | `mkd` | Macedonian | 78.0% |
-| Cyrillic | `mon` | Mongolian | — |
-| Cyrillic | `myv` | Erzya | — |
-| Cyrillic | `rus` | Russian | 73.1% |
-| Cyrillic | `sah` | Yakut (Sakha) | — |
-| Cyrillic | `srp` | Serbian | 82.1% |
-| Cyrillic | `tat` | Tatar | 84.1% |
-| Cyrillic | `tgk` | Tajik | 91.4% |
-| Cyrillic | `ukr` | Ukrainian | 81.5% |
-| Devanagari | `hin` | Hindi | 72.0% |
-| Devanagari | `mar` | Marathi | 75.4% |
-| Devanagari | `nep` | Nepali | — |
-| Ethiopic | `amh` | Amharic | 100% |
-| Georgian | `kat` | Georgian | 100% |
-| Greek | `ell` | Greek | 100% |
-| Gujarati | `guj` | Gujarati | 100% |
-| Gurmukhi | `pan` | Punjabi | 100% |
-| Hangul | `kor` | Korean | 100% |
-| Hebrew | `heb` | Hebrew | 92.6% |
-| Hebrew | `ydd` | Yiddish | 94.0% |
-| Japanese | `jpn` | Japanese | ~100% |
-| Kannada | `kan` | Kannada | 100% |
-| Khmer | `khm` | Khmer | 100% |
-| Lao | `lao` | Lao | 100% |
-| Malayalam | `mal` | Malayalam | 100% |
-| Myanmar | `mya` | Burmese | ~100% |
-| OL Chiki | `sat` | Santali | 100% |
-| Sinhala | `sin` | Sinhala | 100% |
-| Tamil | `tam` | Tamil | 100% |
-| Telugu | `tel` | Telugu | 100% |
-| Thai | `tha` | Thai | 100% |
-
-**Non-Latin subtotal: 49 languages**
-
-Note: `yue` (Cantonese) has near-zero FLORES F1 because the FLORES test set
-contains only Mandarin for Han script — this is an evaluation artefact, not a
-model failure. `yue` and `zho` form a confusable group; the model returns
-whichever scores higher.
-
-`rus` and `hin` are below the 75% threshold but are included
-because their Cyrillic/Devanagari scripts make them far easier to detect at
-short text than the Latin cluster numbers suggest — their confusion in v7 is
-with other Cyrillic/Devanagari languages (which are all retained here, keeping
-the within-family competition intact).
-
----
-
-### Latin-script languages
-
-#### Major European (34)
-
-All are included unconditionally. Those currently
-below 75% F1@20 in v7 are expected to recover once their primary confusers
-(listed in the exclusions section) are removed from the model.
-
-| Code | Language | v7 F1@20 | Notes |
-|------|----------|----------|-------|
-| `afr` | Afrikaans | 73.2% | |
-| `cat` | Catalan | 66.3% | Recovers when `oci`, `glg`, `ast` removed |
-| `ces` | Czech | 77.9% | |
-| `cym` | Welsh | 100% | Distinctive digraph orthography |
-| `dan` | Danish | 57.6% | Recovers when Scandinavian cluster thins |
-| `deu` | German | 68.6% | Recovers when `gsw`, `bar`, `nds`, `ltz` removed |
-| `eng` | English | 45.5% | Recovers when many Latin-script confusers removed |
-| `est` | Estonian | 78.7% | |
-| `eus` | Basque | 78.3% | Language isolate; distinctive orthography |
-| `fao` | Faroese | 75.4% | |
-| `fin` | Finnish | 72.6% | Agglutinative; distinctive character profile |
-| `fra` | French | 66.2% | Recovers when `oci`, `wln`, `pms`, `bre` removed |
-| `gla` | Scottish Gaelic | 88.0% | |
-| `gle` | Irish | 90.6% | |
-| `glg` | Galician | 61.2% | Included; major co-official language of Spain |
-| `hrv` | Croatian | 72.8% | |
-| `hun` | Hungarian | 83.7% | |
-| `isl` | Icelandic | 81.5% | |
-| `ita` | Italian | 62.9% | Recovers when `cos`, `lmo`, `nap`, `scn`, `srd`, `fur`, `lij` removed |
-| `lav` | Latvian | — | EU official language; no FLORES eval data |
-| `lit` | Lithuanian | 83.6% | |
-| `ltz` | Luxembourgish | 73.3% | Included; official language of Luxembourg |
-| `mlt` | Maltese | 72.6% | Semitic vocabulary in Latin orthography |
-| `nld` | Dutch | 47.4% | Recovers when `vls`, `lim`, `nds`, `afk` reduce competition |
-| `nno` | Norwegian Nynorsk | 60.5% | Included alongside `nob` |
-| `nob` | Norwegian Bokmål | 49.7% | |
-| `pol` | Polish | 73.4% | |
-| `por` | Portuguese | 66.8% | |
-| `ron` | Romanian | 83.3% | |
-| `slk` | Slovak | 79.1% | |
-| `slv` | Slovenian | 74.2% | |
-| `spa` | Spanish | 64.7% | Recovers when `ast`, `oci`, `glg` reduce competition |
-| `swe` | Swedish | 56.5% | |
-| `tur` | Turkish | 72.2% | |
-
-#### African Latin-script (16)
-
-| Code | Language | v7 F1@20 |
-|------|----------|----------|
-| `hau` | Hausa | 84.2% |
-| `ibo` | Igbo | 89.2% |
-| `kab` | Kabyle | 80.3% |
-| `kin` | Kinyarwanda | 80.6% |
-| `lin` | Lingala | — |
-| `lug` | Luganda | 84.1% |
-| `nso` | Northern Sotho | 76.8% |
-| `orm` | Oromo | — |
-| `sna` | Shona | 80.1% |
-| `som` | Somali | 85.0% |
-| `swh` | Swahili | 77.6% |
-| `tum` | Tumbuka | 79.5% |
-| `twi` | Twi | 85.1% |
-| `wol` | Wolof | — |
-| `yor` | Yoruba | 83.9% |
-| `zul` | Zulu | 83.7% |
-
-#### Asian / Pacific / Americas (13)
-
-| Code | Language | v7 F1@20 |
-|------|----------|----------|
-| `ace` | Acehnese | 75.8% |
-| `bjn` | Banjar | 54.0% |
-| `grn` | Guaraní | 81.1% |
-| `ilo` | Ilokano | 79.9% |
-| `ind` | Indonesian | 35.3% (confusable group with `msa`) |
-| `jav` | Javanese | 58.5% |
-| `lus` | Mizo | 80.9% |
-| `min` | Minangkabau | 61.0% |
-| `msa` | Malay | — (confusable group with `ind`) |
-| `sun` | Sundanese | 62.1% |
-| `tgl` | Tagalog | 81.2% |
-| `tuk` | Turkmen | 79.8% |
-| `vie` | Vietnamese | 93.6% |
-
-#### Other Latin-script (5)
-
-| Code | Language | v7 F1@20 | Notes |
-|------|----------|----------|-------|
-| `aze` | Azerbaijani (North) | — | Latin-script; distinct from `azb` (Arabic-script South Azerbaijani) |
-| `ban` | Balinese | — | |
-| `pap` | Papiamento | 69.1% | |
-| `uzb` | Uzbek | — | |
-| `war` | Waray | 75.5% | |
-
-**Latin subtotal: 68 languages**
-
-**Grand total: 122 languages** (v1, 2026-03-10; excludes `ceb`)
-
----
-
-## Excluded languages (from v7 model)
-
-These languages appear in the v7 203-language model but are omitted from the
-short-text model. In each case the exclusion is expected to improve detection
-of a closely related majority language.
-
-### Gallo-Romance cluster (harming French and Italian)
-
-Removing these languages is the single largest lever for improving French and
-Italian detection at short text. Their character n-gram profiles at 20–50
-chars are nearly indistinguishable from their parent languages.
-
-| Code | Language | Primary harm |
-|------|----------|-------------|
-| `oci` | Occitan | French, Catalan, Spanish |
-| `cos` | Corsican | Italian |
-| `lmo` | Lombard | Italian |
-| `lij` | Ligurian | Italian |
-| `fur` | Friulian | Italian |
-| `scn` | Sicilian | Italian |
-| `srd` | Sardinian | Italian |
-| `wln` | Walloon | French |
-| `pms` | Piedmontese | Italian, French |
-| `nap` | Neapolitan | Italian |
-
-### Ibero-Romance cluster (harming Spanish and Portuguese)
-
-| Code | Language | Primary harm |
-|------|----------|-------------|
-| `ast` | Asturian | Spanish, Portuguese |
-| `ext` | Extremaduran | Spanish |
-| `arg` | Aragonese | Spanish |
-
-### Germanic cluster (harming German, Dutch, and English)
-
-| Code | Language | Primary harm |
-|------|----------|-------------|
-| `nds` | Low Saxon | German, Dutch, English |
-| `bar` | Bavarian | German |
-| `gsw` | Alsatian | German |
-| `vls` | West Flemish | Dutch |
-| `lim` | Limburgish | Dutch |
-| `fry` | Western Frisian | Dutch, English |
-| `nds-nl` | Low Saxon Dutch | Dutch (also excluded from v5) |
-| `bre` | Breton | French |
-
-Note: `ltz` (Luxembourgish) is **retained** — it is an official national
-language and distinct enough in practice to be worth the minor German bleed.
-
-### Slavic cluster (high mutual confusion at 20 chars)
-
-The South Slavic and West Slavic languages are already known to be mutually
-confusable at short text. The short-text model retains only the national
-standard languages.
-
-| Code | Language | Primary harm |
-|------|----------|-------------|
-| `hsb` | Upper Sorbian | Czech, Polish |
-| `dsb` | Lower Sorbian | Czech, Polish |
-| `rue` | Rusyn | Ukrainian, Slovak |
-| `szl` | Silesian | Polish |
-| `csb` | Kashubian | Polish |
-| `sgs` | Samogitian | Lithuanian |
-
-### Classical and constructed languages
-
-| Code | Language | Reason |
-|------|----------|--------|
-| `san` | Sanskrit | Classical; Devanagari profile overlaps hin/mar/nep |
-| `epo` | Esperanto | Constructed; Latin profile overlaps many Romance languages |
-| `lat` | Latin | Classical; Latin profile overlaps Romance |
-| `grc` | Ancient Greek | Classical |
-| `tlh` | Klingon | Constructed |
-
-### Other excluded (corpus quality or low practical value)
-
-| Code | Language | Reason |
-|------|----------|--------|
-| `ceb` | Cebuano | FLORES @20 = 26.9% across all feature configs in March 2026 ablation — below useful threshold. Heavily confuses with `tgl` (Tagalog); dropping `ceb` is expected to recover `tgl` significantly. Note: Wikipedia Cebuano corpus was bot-generated stubs (excluded from general model for the same reason); MADLAD-400 Cebuano data (Kudugunta et al., arXiv:2309.04662) was used in the short model but was insufficient to separate from Tagalog at 20 chars. |
-| `glv` | Manx | ~2,000 speakers; causes English confusion |
-| `cor` | Cornish | Very small speaker base; causes English/Welsh confusion |
-| `haw` | Hawaiian | Small corpus; Latin profile at short text |
-| `lzh` | Classical Chinese | Classical; HAN script same as `zho` |
-
----
-
-## Notes on confusable pairs retained
-
-The following confusable-group pairs from v7 are carried over unchanged into
-the short-text model. The model returns whichever member scores higher.
-
-| Pair | Notes |
-|------|-------|
-| `ind` / `msa` | Indonesian / Malay. Near-identical written forms; both important for SE Asian document processing. |
-| `yue` / `zho` | Cantonese / Mandarin. Han script shared; model returns higher-scoring member. |
-| `xho` / `zul` | Xhosa / Zulu. Both included; confusable group unchanged. |
-
----
-
-## Post-training results (v1, 2026-03-10)
-
-Evaluation on FLORES-200 dev split with `--strategy SHORT_TEXT`. Per-language
-F1@20 for the European languages that motivated the short-text model:
-
-| Language | v7 std @20 | Short v1 @20 | Target | Status |
-|----------|:---:|:---:|:---:|--------|
-| English (`eng`) | 45.5% | 52.4% | ≥ 85% | **Not met.** Primary issue is low confidence (`unk`), not misclassification. At 20 chars, English function words are shared across many Latin-script languages. See `flores200-dev-eval.md` for confusion analysis. Planned fix: stop-word anchoring. |
-| French (`fra`) | 66.2% | 74.8% | ≥ 85% | Improved +8.6 pp. Removing Gallo-Romance cluster helped. |
-| German (`deu`) | 68.6% | 76.4% | ≥ 85% | Improved +7.8 pp. Removing `gsw`/`bar`/`nds` helped. |
-| Italian (`ita`) | 62.9% | 69.0% | ≥ 85% | Improved +6.1 pp. Still below target; `cos`/`lmo` etc. are gone but short-text Romance confusion remains. |
-| Spanish (`spa`) | 64.7% | 71.6% | ≥ 85% | Improved +6.9 pp. |
-| Dutch (`nld`) | 47.4% | 71.1% | ≥ 85% | Improved +23.7 pp — largest gain. Removing `vls`/`lim`/`fry` was effective. |
-| Portuguese (`por`) | 66.8% | 79.6% | ≥ 85% | Improved +12.8 pp. |
-
-**Macro-F1 @20 (coverage-adjusted):** 84.68% over 122 languages, vs 78.71%
-for the v7 standard model over 203 languages.
-
-No European language met the 85% @20 target. The gains are substantial
-(+6 to +24 pp) but 20 characters remains fundamentally difficult for
-Latin-script languages. The targets were aspirational; the actual results are
-consistent with the model performing well at 50+ chars. At @50 the short-text
-model achieves 96.07% macro-F1.
-
-**Indonesian (`ind`) F1@20:** confusable with `msa` by design; group accuracy
-is the right metric, not per-language F1.
diff --git a/docs/modules/ROOT/pages/advanced/lang-detection/supported-languages.md b/docs/modules/ROOT/pages/advanced/lang-detection/supported-languages.md
deleted file mode 100644
index 07e6081c09..0000000000
--- a/docs/modules/ROOT/pages/advanced/lang-detection/supported-languages.md
+++ /dev/null
@@ -1,240 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Supported Languages
-
-Languages supported by the CharSoup language detector, extracted from the
-model binaries. The model binary is the source of truth; this list is for
-human reference.
-
-To query programmatically:
-
-```java
-CharSoupLanguageDetector.getSupportedLanguages(Strategy.STANDARD) // 203 languages
-CharSoupLanguageDetector.getSupportedLanguages(Strategy.SHORT_TEXT) // 122 languages
-```
-
-- **Standard model**: `langdetect-v7-20260306.bin` (3.2 MB, 16 384 buckets, flags 0x075)
-- **Short-text model**: `langdetect-short-v1-20260310.bin` (3.8 MB, 32 768 buckets, flags 0x0a1) — optimized for 20–50 char inputs; see `short-text-language-decisions.md` for inclusion/exclusion rationale.
-
-| Code | Std | Short |
-|------|:---:|:-----:|
-| `ace` | x | |
-| `afr` | x | x |
-| `aka` | x | x |
-| `alt` | x | |
-| `amh` | x | x |
-| `ami` | x | |
-| `ara` | x | x |
-| `arg` | x | |
-| `asm` | x | x |
-| `ava` | x | |
-| `avk` | x | |
-| `azb` | x | x |
-| `aze` | x | x |
-| `bak` | x | x |
-| `ban` | x | |
-| `bar` | x | |
-| `bcl` | x | |
-| `be-x-old` | x | |
-| `bel` | x | x |
-| `ben` | x | x |
-| `bjn` | x | |
-| `bre` | x | |
-| `bul` | x | x |
-| `bxr` | x | |
-| `cat` | x | x |
-| `cdo-x-rom` | x | x |
-| `ceb` | x | |
-| `ces` | x | x |
-| `che` | x | x |
-| `chv` | x | x |
-| `ckb` | x | x |
-| `cnh` | x | |
-| `cor` | x | |
-| `cos` | x | |
-| `csb` | x | |
-| `cym` | x | x |
-| `dag` | x | x |
-| `dan` | x | x |
-| `deu` | x | x |
-| `diq` | x | |
-| `div` | x | x |
-| `dsb` | x | |
-| `ell` | x | x |
-| `eng` | x | x |
-| `epo` | x | |
-| `est` | x | x |
-| `eus` | x | x |
-| `ewe` | x | x |
-| `ext` | x | |
-| `fao` | x | |
-| `fas` | x | x |
-| `fin` | x | x |
-| `fra` | x | x |
-| `frr` | x | |
-| `fry` | x | x |
-| `gla` | x | |
-| `gle` | x | x |
-| `glg` | x | |
-| `glv` | x | |
-| `gom` | x | |
-| `grn` | x | |
-| `gsw` | x | |
-| `guj` | x | x |
-| `hak-x-rom` | x | |
-| `hau` | x | x |
-| `heb` | x | x |
-| `hil` | x | x |
-| `hin` | x | x |
-| `hrv` | x | x |
-| `hsb` | x | |
-| `hun` | x | x |
-| `hye` | x | x |
-| `hyw` | x | x |
-| `ibo` | x | x |
-| `ido` | x | |
-| `ile` | x | |
-| `ilo` | x | |
-| `ina` | x | |
-| `ind` | x | x |
-| `isl` | x | x |
-| `ita` | x | x |
-| `jav` | x | x |
-| `jbo` | x | |
-| `jpn` | x | x |
-| `kaa` | x | |
-| `kab` | x | |
-| `kan` | x | x |
-| `kat` | x | x |
-| `kaz` | x | x |
-| `kha` | x | x |
-| `khm` | x | x |
-| `kin` | x | x |
-| `kir` | x | x |
-| `kor` | x | x |
-| `kpv` | x | |
-| `ksh` | x | |
-| `kur` | x | x |
-| `lao` | x | x |
-| `lat` | x | |
-| `lav` | x | x |
-| `lez` | x | |
-| `lfn` | x | |
-| `lim` | x | |
-| `lit` | x | x |
-| `ltz` | x | |
-| `lug` | x | x |
-| `lus` | x | x |
-| `mal` | x | x |
-| `mar` | x | x |
-| `mhr` | x | |
-| `min` | x | |
-| `mkd` | x | x |
-| `mlg` | x | x |
-| `mlt` | x | x |
-| `mon` | x | x |
-| `mrj` | x | |
-| `msa` | x | |
-| `mwl` | x | |
-| `mya` | x | x |
-| `myv` | x | |
-| `mzn` | x | |
-| `nds` | x | |
-| `nep` | x | x |
-| `nld` | x | x |
-| `nno` | x | |
-| `nob` | x | x |
-| `nqo` | x | x |
-| `nso` | x | |
-| `nya` | x | x |
-| `olo` | x | |
-| `ori` | x | x |
-| `orm` | x | x |
-| `oss` | x | x |
-| `pam` | x | |
-| `pan` | x | x |
-| `pap` | x | |
-| `pfl` | x | |
-| `pnb` | x | |
-| `pol` | x | x |
-| `por` | x | x |
-| `pus` | x | x |
-| `roh` | x | |
-| `ron` | x | x |
-| `rue` | x | |
-| `rus` | x | x |
-| `sah` | x | x |
-| `san` | x | |
-| `sat` | x | x |
-| `sgs` | x | |
-| `sin` | x | x |
-| `skr` | x | |
-| `slk` | x | x |
-| `slv` | x | x |
-| `sme` | x | |
-| `smn` | x | |
-| `smo` | x | x |
-| `sna` | x | x |
-| `snd` | x | x |
-| `som` | x | x |
-| `spa` | x | x |
-| `sqi` | x | x |
-| `srp` | x | x |
-| `stq` | x | |
-| `sun` | x | x |
-| `swe` | x | x |
-| `swh` | x | x |
-| `szl` | x | |
-| `szy` | x | |
-| `tam` | x | x |
-| `tat` | x | x |
-| `tay` | x | x |
-| `tel` | x | x |
-| `tet` | x | x |
-| `tgk` | x | x |
-| `tgl` | x | x |
-| `tha` | x | x |
-| `tir` | x | x |
-| `trv` | x | x |
-| `tsn` | x | x |
-| `tso` | x | x |
-| `tuk` | x | x |
-| `tum` | x | |
-| `tur` | x | x |
-| `tyv` | x | |
-| `udm` | x | x |
-| `uig` | x | x |
-| `ukr` | x | x |
-| `urd` | x | x |
-| `uzb` | x | x |
-| `vep` | x | |
-| `vie` | x | x |
-| `vls` | x | |
-| `vol` | x | |
-| `vro` | x | |
-| `war` | x | |
-| `wln` | x | |
-| `xho` | x | x |
-| `xmf` | x | |
-| `ydd` | x | x |
-| `yor` | x | x |
-| `yue` | x | |
-| `zho` | x | x |
-| `zul` | x | x |
-
-**Totals**: 203 standard, 122 short-text (all short-text languages are a subset of standard).
diff --git a/docs/modules/ROOT/pages/advanced/language-detection-build.adoc b/docs/modules/ROOT/pages/advanced/language-detection-build.adoc
index 1b999784bc..750a9c3073 100644
--- a/docs/modules/ROOT/pages/advanced/language-detection-build.adoc
+++ b/docs/modules/ROOT/pages/advanced/language-detection-build.adoc
@@ -17,63 +17,48 @@
= Building the CharSoup Language Detector
-This page documents how the `tika-langdetect-charsoup` language detection model
-is trained, the decisions made along the way, and benchmark comparisons against
-the existing OpenNLP-based detector. For architecture and API details, see
+How the `tika-langdetect-charsoup` model is trained, the decisions baked into it,
+and how it benchmarks against the alternatives. For architecture and API, see
xref:advanced/language-detection.adoc[Language Detection].
+Corpus paths below are written as `<corpus>` and `<workdir>`; substitute your own.
+
== Training Corpus
The primary training data comes from Wikipedia database dumps
-(https://dumps.wikimedia.org/[dumps.wikimedia.org]). Wikipedia is preferred
-over web-crawl corpora for quality: articles are human-authored, editorial
-standards filter boilerplate and spam, and the sentence distribution reflects
-genuine prose rather than SEO content or duplicated web templates.
-
-Sentences are extracted from the Wikipedia XML dumps using the
-`extract_wiki_sentences.py` script, which strips markup, splits into sentences,
-and writes one `lineNum<TAB>sentence` file per language directory:
+(https://dumps.wikimedia.org/[dumps.wikimedia.org]). Wikipedia is preferred over
+web-crawl corpora for quality: articles are human-authored, editorial standards
+filter boilerplate and spam, and the sentence distribution reflects genuine prose
+rather than SEO content or duplicated web templates.
+
+`extract_wiki_sentences.py` strips markup, splits into sentences, and writes one
+`lineNum<TAB>sentence` file per language directory. For 17 languages with thin
+Wikipedia coverage, `extract_madlad_to_wiki.py` writes a parallel
+`sentences_madlad.txt` from
+https://huggingface.co/datasets/allenai/MADLAD-400[MADLAD-400] (Magnusson et al.,
+2023), applying the same quality filters and capping at 500,000 sentences per
+language. `PrepareCorpus` reads every `*.txt` in a language directory:
----
-~/datasets/wikipedia-dumps/
+<corpus>/wikipedia-dumps/
eng/sentences.txt
deu/sentences.txt
- ara/sentences.txt
- ...
-----
-
-For 16 languages with insufficient Wikipedia coverage, sentences are
-supplemented from
-https://huggingface.co/datasets/allenai/MADLAD-400[MADLAD-400]
-(Magnusson et al., 2023). These languages are written into a parallel
-`sentences_madlad.txt` file alongside the Wikipedia data, and `PrepareCorpus`
-reads all `*.txt` files from each language directory automatically:
-
-----
-~/datasets/wikipedia-dumps/
mya/
sentences.txt (Wikipedia)
sentences_madlad.txt (MADLAD supplement)
- xho/
- sentences.txt
- sentences_madlad.txt
...
----
-The MADLAD-supplemented languages are:
-`mya`, `xho`, `nya`, `smo`, `sot`, `tet`, `orm`, `udm`, `tir`, `hil`,
-`ewe`, `tso`, `aka`, `tsn`, `ceb`, `mlg`, `che`.
+MADLAD-supplemented languages: `mya`, `xho`, `nya`, `smo`, `sot`, `tet`, `orm`,
+`udm`, `tir`, `hil`, `ewe`, `tso`, `aka`, `tsn`, `ceb`, `mlg`, `che`.
-The `extract_madlad_to_wiki.py` script handles extraction from MADLAD's
-document-per-line format (paragraph boundaries encoded as literal `\n`
-sequences), applies quality filters identical to those used by the main
-download pipeline, and caps output at 500,000 sentences per language.
+MADLAD documents encode paragraph boundaries as literal `\n` escape sequences;
+the extractor splits on those before processing.
=== Deduplication
-Deduplication is performed at the Java training-pipeline stage using FNV-1a
-64-bit hashing. Web-crawl data has lower duplication rates than Leipzig news
-corpora, so a single deduplication pass is sufficient.
+Deduplication happens in the Java training pipeline using FNV-1a 64-bit hashing.
+One pass is enough — this corpus has a low duplication rate.
[[_language_code_merging]]
=== Language Code Merging
@@ -98,67 +83,59 @@ download and training:
| `swa` (Swahili macrolanguage) | `swh` (Coastal Swahili) | Code variant
| `yid` (Yiddish macrolanguage) | `ydd` (Eastern Yiddish) | Code variant
| `zsm` (Standard Malay) | `msa` (Malay) | Code variant
+| `zza` (Zazaki) | `diq` (Dimli) | Code variant
|===
-The same merge map is maintained in three places and must be kept in sync:
+The same merge map is copied into four places and must be kept in sync:
+* `CorpusAliases.java` — `LANG_MERGE_MAP`, the canonical definition
+* `CommonTokenGenerator.java` — `LANG_MERGE_MAP`, a hand-maintained copy in
+ tika-eval-core (test-scope classes cannot be shared across modules)
* `download_madlad.py` — `LANG_MERGE_MAP`
-* `PrepareCorpus.java` — `LANG_MERGE_MAP`
-* `CommonTokenGenerator.java` — `LANG_MERGE_MAP`
+* `download_corpus.py` — `LANG_MERGE_MAP`
+
+The two Python copies currently omit `zza` → `diq`.
=== Corpus Cleaning
-Two data-quality issues were identified during development and addressed in
-`CorpusReader.java`:
-
-**Breton (`bre`) noise** — approximately 5% of MADLAD Breton sentences were
-French blog posts, identifiable by lines containing three or more consecutive
-tildes (`~~~`), a Common Crawl redaction marker. A filter discards any sentence
-containing this pattern.
-
-**Dhivehi (`div`) mixed-script headlines** — MADLAD Dhivehi documents
-consistently begin with a Latin-script headline followed by Thaana-script body
-text, separated by a literal `\n` escape sequence. The pipeline splits on this
-separator and treats each segment as a distinct sentence, preventing the Latin
-headline from polluting the Thaana training signal. This raised 20-character
-accuracy from 32.9% to 95.5%.
-
-=== Filtering Low-Resource Languages
-
-Languages with fewer than **10,000 sentences** after deduplication are excluded.
-This threshold ensures enough data for the model to learn useful distributions
-even after the mislabel-filtering step.
-
-=== Explicit Language Exclusions
-
-Some languages meet the 10,000-sentence minimum but are explicitly excluded.
-Exclusion decisions are made after a full training run by evaluating per-language
-F1 on the held-out test set and inspecting confusion patterns. A language is
-excluded if it falls into one or more of these categories:
-
-**Accuracy interference with a closely related language.**
-When a language's written form is nearly identical to a more widely-used language
-at the character n-gram level, including both causes the model to split
-probability mass between them. This depresses accuracy for the more widely-used
-language by an amount that exceeds any benefit from detecting the variant. The
-exclusion is a deliberate choice to serve the larger user population; it does not
-reflect on the importance or validity of the excluded language.
-
-**Own accuracy below a useful threshold.**
-If the model cannot achieve reliable accuracy for a language — because its
-character profile overlaps too heavily with other included languages — then
-returning a prediction for it does more harm than good. A confidently wrong
-prediction is worse than no prediction.
-
-**Training corpus not representative of natural language use.**
-Some languages have MADLAD corpus entries dominated by boilerplate, Lorem Ipsum
-placeholder text, or other non-natural content. A model trained on such data
-learns the boilerplate rather than the language, producing unreliable results in
-real-world documents.
-
-These exclusions are applied by removing the language's pool file before Pass 2
-training. If future corpus improvements make a language reliably distinguishable,
-it can be reintroduced by adding its pool file back and retraining.
+`clean_madlad.py` applies the generic filters before a sentence ever reaches the
+Java pipeline: length (50–500 characters), alphabetic ratio (≥ 0.5), a
+dirty-pattern regex (URLs, wiki markup, ISBN/ISSN, Lorem ipsum), a fastText
+top-10 contamination filter, and exact dedup against the language's existing
+`sentences.txt`.
+
+`PrepareCorpus.java` then enforces script consistency, but only for the languages
+listed in its `SCRIPT_CONSISTENCY_LANGS` map: a sentence is dropped unless at
+least `SCRIPT_CONSISTENCY_THRESHOLD` (0.80) of its letters belong to that
+language's expected script. Two per-language carve-outs:
+
+**Dhivehi (`div`)** — Maldivian prose naturally mixes Thaana with Arabic script
+for Islamic phrases, so 0.80 cuts too aggressively; the threshold is overridden
+to 0.50.
+
+**Japanese and Korean (`jpn`, `kor`)** — legitimately mix some Latin, so instead
+of the script-consistency rule they are capped by `MAX_LATIN_RATIO` (0.50).
+
+=== Language Exclusions
+
+Languages with fewer than **10,000 sentences** after deduplication are dropped —
+below that there is not enough data to survive mislabel filtering.
+
+Some languages clear that bar and are still excluded, by removing the pool file
+before Pass 2. The decision is made after a full training run, from per-language
+F1 on the held-out test set and its confusion pattern, on any of three grounds:
+
+* **Interference with a closely related language.** When two written forms are
+ nearly identical at the character n-gram level, including both splits
+ probability mass and costs the more widely-used language more than the variant
+ gains. This serves the larger user population; it says nothing about the
+ importance of the excluded language.
+* **Own accuracy too low to be useful.** A confidently wrong prediction is worse
+ than no prediction.
+* **Corpus not representative.** Where MADLAD entries are dominated by
+ boilerplate or Lorem Ipsum, the model learns the boilerplate.
+
+Any of these is reversible: restore the pool file and retrain if the corpus improves.
== Data Splitting Strategy
@@ -167,11 +144,11 @@ it can be reintroduced by adding its pool file back and retraining.
| Split | Size | Preprocessing
| **Test**
-| 10% per language (max 20,000)
+| 10% per language, capped by `--max-test` (default 2,000)
| Raw (no preprocessing)
| **Dev**
-| 10% per language (max 20,000)
+| 10% per language, capped by `--max-dev` (default 2,000)
| Preprocessed (NFC, lowercase, URL/email stripped)
| **Training pool**
@@ -179,6 +156,11 @@ it can be reintroduced by adding its pool file back and retraining.
| Preprocessed, stored as per-language files
|===
+The two caps default differently depending on which tool does the prep:
+`PrepareCorpus` defaults both to 2,000, `TrainLanguageModel`'s own prep step
+defaults both to 20,000. The 20260320 build ran `PrepareCorpus` separately, so
+2,000 applied.
+
Each epoch draws a fresh sample from the pool: binary-search finds a flat cap C
such that `Σ min(n_i, C) ≈ 5,000,000`, then each language contributes up to C
sentences, globally shuffled. High-resource languages are capped per epoch;
@@ -188,8 +170,9 @@ low-resource languages contribute all their data every epoch.
=== Pass 1: Initial Training
-AdamW for 2 epochs followed by Hogwild! SGD for up to 3 more epochs, each
-with epoch-level resampling from the full training pool.
+AdamW for 2 epochs followed by Hogwild! SGD for up to 6 more (8 epochs total,
+subject to early stopping), each with epoch-level resampling from the full
+training pool.
=== Mislabeled Sentence Filtering
@@ -219,16 +202,21 @@ demonstrably confuses them at meaningful rates on held-out data. Groups are not
added speculatively; each entry is backed by observed confusion in evaluation.
Groups are defined in
-`tika-langdetect-charsoup-core/src/main/resources/.../confusables.txt`
+`tika-langdetect-charsoup/src/main/resources/org/apache/tika/langdetect/charsoup/confusables.txt`
and must only contain codes that are actual output classes of the trained model
(i.e., present in the training corpus). Dead codes add no benefit.
Current groups:
* `msa` / `ind` — Malay and Indonesian share vocabulary and script so heavily
- that in-distribution confusion exceeds 20% in both directions.
+ that they cross-predict at roughly 9% at 500 characters.
* `xho` / `zul` — Xhosa and Zulu are both Nguni Bantu languages written in the
same Latin-based orthography with very similar character n-gram profiles.
+* `bel` / `be-x-old` — the two written standards for Belarusian; grouped to keep
+ training-time contamination filtering from playing them off against each other.
+* `yue` / `zho` — Cantonese and Mandarin share Han script, so their character
+ n-gram profiles overlap heavily. FLORES F1 for `yue` is near zero because
+ FLORES carries only Mandarin text for that script.
These groups are used in:
@@ -259,8 +247,8 @@ tika-eval/tika-eval-core/target/classes:\
tika-langdetect/tika-langdetect-charsoup/target/classes:\
tika-langdetect/tika-langdetect-charsoup-core/target/classes \
org.apache.tika.eval.core.tokens.tools.CommonTokenGenerator \
- ~/datasets/madlad/data \
- /tmp/common_tokens_new \
+ <corpus>/madlad/data \
+ <workdir>/common-tokens \
30000 10 \
--model tika-langdetect/tika-langdetect-charsoup-core/src/main/resources/org/apache/tika/langdetect/charsoup/langdetect-20260320.bin
----
@@ -277,14 +265,13 @@ each language subdirectory.
== Current Build: 20260320
-The model `langdetect-20260320.bin` is the current production model (trained
-2026-03-20). It uses `SaltedNgramFeatureExtractor` with trigrams, 4-grams,
-script block features, L2 normalization, and short-word-anchored word bigrams.
+`langdetect-20260320.bin` is the shipped model. Its output classes are listed in
+xref:advanced/charsoup-supported-languages.adoc[CharSoup Supported Languages].
=== 20260320 Training Configuration
* **Corpus:** Wikipedia dumps + MADLAD-400 supplements
-* **Languages:** 204 (includes Tibetan `bod`)
+* **Languages:** 204
* **Pool cap:** 500,000 sentences per language
* **Feature extractor:** `SaltedNgramFeatureExtractor` — positional-salted
character bigrams (BOW/EOW/FULL_WORD/MID), trigrams, 4-grams, CJK character
@@ -295,7 +282,8 @@ script block features, L2 normalization, and short-word-anchored word bigrams.
* **Target epoch total:** 5,000,000 sentences per epoch
* **Two-pass training:** Pass 1 on full pool → mislabel filter → Pass 2
* **JVM:** `-Xmx8g`
-* **Model size:** ~6.4 MB on disk, ~8.1 MB heap (INT8 quantized)
+* **Model size:** ~6.4 MB on disk (INT8 quantized); the ~8.1 MB in the eval log is
+ the used-heap delta measured across the load call, not the retained size
=== 20260320 Corpus Preparation
@@ -308,8 +296,8 @@ script block features, L2 normalization, and short-word-anchored word bigrams.
./mvnw -pl tika-langdetect/tika-langdetect-charsoup exec:java \
-Dexec.mainClass="org.apache.tika.langdetect.charsoup.tools.PrepareCorpus" \
-Dexec.classpathScope=test \
- -Dexec.args="--corpus ~/datasets/wikipedia-dumps \
- --output-dir ~/datasets/wikipedia-model-20260320/preprocessed \
+ -Dexec.args="--corpus <corpus>/wikipedia-dumps \
+ --output-dir <workdir>/model-20260320/preprocessed \
--max-train 500000" \
-DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true
----
@@ -321,9 +309,9 @@ script block features, L2 normalization, and short-word-anchored word bigrams.
./mvnw -pl tika-langdetect/tika-langdetect-charsoup exec:java \
-Dexec.mainClass="org.apache.tika.langdetect.charsoup.tools.TrainLanguageModel" \
-Dexec.classpathScope=test \
- -Dexec.args="--corpus ~/datasets/wikipedia-dumps \
- --prep-dir ~/datasets/wikipedia-model-20260320/preprocessed \
- --output ~/datasets/wikipedia-model-20260320/langdetect-20260320.bin \
+ -Dexec.args="--corpus <corpus>/wikipedia-dumps \
+ --prep-dir <workdir>/model-20260320/preprocessed \
+ --output <workdir>/model-20260320/langdetect-20260320.bin \
--buckets 32768 \
--4grams --salted --l2-norm --word-bigrams" \
-DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true \
@@ -334,7 +322,7 @@ Copy the trained model into the resources directory:
[source,bash]
----
-cp ~/datasets/wikipedia-model-20260320/langdetect-20260320.bin \
+cp <workdir>/model-20260320/langdetect-20260320.bin \
tika-langdetect/tika-langdetect-charsoup-core/src/main/resources/\
org/apache/tika/langdetect/charsoup/langdetect-20260320.bin
----
@@ -342,88 +330,61 @@ org/apache/tika/langdetect/charsoup/langdetect-20260320.bin
Then update `MODEL_RESOURCE` in `CharSoupLanguageDetector` to point to the
new file, and remove the old binary.
-=== 20260320 Changes from previous builds
-
-* **32,768 buckets** (up from 16,384) — doubles the feature space, reducing
- hash collisions at the cost of doubling model size. Net gain at all lengths.
-* **4-grams added** (`--4grams`) — improves disambiguation of short text.
-* **ScriptCategory COUNT reverted to 24** — the COUNT=36 expansion (adding 12
- Indic script categories) was net negative. Indic languages were already 98–99%+
- accurate from n-grams alone; the extra categories added bucket collision noise
- for Latin/Cyrillic languages without improving Indic accuracy. Reverted to 24.
-* **Log-dampening of script features reverted** — replacing raw character counts
- with `log1p(count)` hurt 108 languages while helping 64. Kept raw counts.
-* **Word bigrams** (`--word-bigrams`) — short-word-anchored bigrams
- (anchor = prev word ≤ 3 chars, e.g. "the X", "de X"). +0.32pp @20 with
- significant wins for minority/confusable languages.
-* **Single model** — the separate short-text model is retired. The unified 32K-bucket
- model outperforms it at all text lengths while covering all 204 languages.
-
=== 20260320 Evaluation: FLORES-200 Dev Set
Results on the https://github.com/facebookresearch/flores[FLORES-200] dev set
-(204 test languages, 997 sentences each). All scores are macro-averaged F1.
+(997 sentences per language, 203,381 test sentences in total). All scores are
+macro-averaged F1.
Raw eval output: link:{attachmentsdir}/advanced/flores-eval-20260320.txt[flores-eval-20260320.txt].
-==== Coverage-adjusted accuracy (each detector on its own supported languages)
+==== Coverage-adjusted accuracy
+
+Each detector scored only on the languages it supports — sentences in languages a
+detector does not cover are skipped for that detector. This flatters narrow
+detectors, so read it alongside the breadth-weighted numbers below.
[cols="1,1,1,1,1"]
|===
-| Length | CharSoup mF1 | OpenNLP mF1 | Lingua mF1 | Optimaize mF1
+| Length | CharSoup (204 langs) | OpenNLP (~105) | Lingua (75) | Optimaize (63)
-| @20 chars | **82.51%** | 74.87% | 76.35% | 84.87%
+| @20 chars | 82.51% | 74.87% | 76.35% | **84.87%**
| @50 chars | **94.44%** | 86.09% | 90.99% | 94.44%
-| @100 chars | 96.98% | 90.25% | 95.43% | **96.51%**
-| @200 chars | 97.45% | 91.11% | 96.23% | **96.75%**
-| full text | 97.46% | 91.12% | 96.25% | **96.76%**
-| Languages | **204** | ~105 | 75 | 63
+| @100 chars | **96.98%** | 90.25% | 95.43% | 96.51%
+| @200 chars | **97.45%** | 91.11% | 96.23% | 96.75%
+| full text | **97.46%** | 91.12% | 96.25% | 96.76%
|===
-NOTE: Coverage-adjusted scores reflect each detector's performance on the
-languages it actually supports. CharSoup covers 204 languages;
-Optimaize covers only 63, Lingua 75, OpenNLP ~105.
+==== Breadth-weighted accuracy
-==== Head-to-head: CharSoup vs OpenNLP (105 shared languages)
+All 203 FLORES languages, with unsupported languages scoring 0 — total useful
+output across all inputs.
[cols="1,1,1,1,1"]
|===
-| Length | CharSoup mF1 | OpenNLP mF1 | CharSoup ms | OpenNLP ms
-
-| @20 chars | **84.23%** | 76.69% | 504 | 228
-| @50 chars | **95.73%** | 87.27% | 431 | 333
-| @100 chars | **98.09%** | 91.05% | 523 | 620
-| @200 chars | **98.51%** | 91.77% | 583 | 969
-| full text | **98.51%** | 91.78% | 604 | 959
-|===
-
-==== Head-to-head: CharSoup vs Lingua (71 shared languages)
+| Length | CharSoup | OpenNLP | Lingua | Optimaize
-[cols="1,1,1,1,1"]
+| @20 chars | **52.43%** | 42.05% | 27.46% | 26.76%
+| @100 chars | **61.63%** | 50.68% | 34.32% | 30.43%
+| full text | **61.93%** | 51.17% | 34.61% | 30.51%
|===
-| Length | CharSoup mF1 | Lingua mF1 | CharSoup ms | Lingua ms
-| @20 chars | **85.28%** | 77.78% | 303 | 2,960
-| @50 chars | **96.37%** | 92.24% | 296 | 5,992
-| @100 chars | 98.51% | **96.58%** | 359 | 9,756
-| @200 chars | 98.82% | **97.37%** | 437 | 12,615
-| full text | 98.83% | **97.38%** | 402 | 12,847
-|===
+==== Head-to-head on shared languages
-==== Head-to-head: CharSoup vs Optimaize (63 shared languages)
+CharSoup versus each alternative, restricted to the languages both support.
-[cols="1,1,1,1,1"]
+[cols="1,2,2,2"]
|===
-| Length | CharSoup mF1 | Optimaize mF1 | CharSoup ms | Optimaize ms
+| Length | vs OpenNLP (105 shared) | vs Lingua (71 shared) | vs Optimaize (63 shared)
-| @20 chars | **86.52%** | 86.30% | 215 | 76
-| @50 chars | **96.92%** | 95.40% | 250 | 367
-| @100 chars | **98.84%** | 97.23% | 320 | 360
-| @200 chars | **99.13%** | 97.34% | 346 | 374
-| full text | **99.14%** | 97.34% | 353 | 385
+| @20 chars | **84.23%** / 76.69% | **85.28%** / 77.78% | **86.52%** / 86.30%
+| @50 chars | **95.73%** / 87.27% | **96.37%** / 92.24% | **96.92%** / 95.40%
+| @100 chars | **98.09%** / 91.05% | **98.51%** / 96.58% | **98.84%** / 97.23%
+| @200 chars | **98.51%** / 91.77% | **98.82%** / 97.37% | **99.13%** / 97.34%
+| full text | **98.51%** / 91.78% | **98.83%** / 97.38% | **99.14%** / 97.34%
|===
-CharSoup matches or leads Optimaize across all lengths on the 63-language
-overlap, while covering 3× more languages total.
+CharSoup leads at every length on every overlap. Optimaize is the closest
+competitor and covers 63 languages to CharSoup's 204.
==== 20260320 Resource Usage
@@ -432,7 +393,7 @@ overlap, while covering 3× more languages total.
| Metric | CharSoup | OpenNLP | Lingua (low accuracy) | Optimaize
| Languages supported | **204** | ~105 | 75 | 63
-| Model heap | **~8.1 MB** | ~79.2 MB | ~0.1 MB | ~94.5 MB
+| Model heap (load-time used-heap delta) | **~8.1 MB** | ~79.2 MB | ~0.1 MB | ~94.5 MB
| Model file (disk) | **6.4 MB** | ~22 MB | — | —
| Throughput (@20) | **~139K sent/s** | ~133K sent/s | ~10K sent/s | ~248K sent/s
| Throughput (full) | **~116K sent/s** | — | — | —
@@ -442,29 +403,6 @@ overlap, while covering 3× more languages total.
NOTE: All detectors evaluated with 12 threads on the FLORES-200 dev set
(203,381 sentences). Throughput is wall-clock sentences per second.
-== Historical Build: v7 (Wikipedia + MADLAD)
-
-v7 used a single general model (`langdetect.bin`) — 203 languages, trained on
-Wikipedia dumps (primary) supplemented by MADLAD for under-resourced languages.
-Uses 16,384 hash buckets and `ScriptAwareFeatureExtractor`. 3.2 MB on disk.
-
-A separate short-text model was also trained for v7 but is retired as of
-20260320; the unified 32K-bucket model supersedes it.
-
-=== v7 General Model Training Configuration
-
-* **Corpus:** Wikipedia dumps (primary) + MADLAD supplement for 17 languages
-* **Languages:** 203
-* **Hash buckets:** 16,384
-* **Feature extractor:** `ScriptAwareFeatureExtractor` — character bigrams,
- trigrams, 3-char suffixes, 3-char prefixes, word unigrams, CJK character unigrams
-* **Target epoch total:** 5,000,000 sentences per epoch
-* **Optimizer:** AdamW (lr=0.001, mini-batch=64) × 2 epochs, then Hogwild!
- SGD (lr=0.01→0.001) × 6 max epochs with within-epoch early stopping
-* **Two-pass training:** Pass 1 on full pool → mislabel filter (removed 1.5%) → Pass 2
-* **Model size:** 3.2 MB (INT8 quantized)
-* **JVM:** `-Xmx8g`
-
== References
For the academic references behind the techniques used in training and inference,
diff --git a/docs/modules/ROOT/pages/advanced/language-detection.adoc b/docs/modules/ROOT/pages/advanced/language-detection.adoc
index b04c61a920..29dd60ca25 100644
--- a/docs/modules/ROOT/pages/advanced/language-detection.adoc
+++ b/docs/modules/ROOT/pages/advanced/language-detection.adoc
@@ -30,49 +30,38 @@ and are loaded automatically via Tika's service discovery.
== Architecture: CharSoupLanguageDetector
-The built-in detector uses a simple but effective architecture based on
-character n-gram language identification (<<cavnar1994>>):
-
-1. **Preprocessing** — truncate, strip URLs/emails, NFC normalize
-2. **Feature extraction** — character n-grams, word unigrams, word suffixes and
- prefixes, with script-aware boundary detection, hashed via FNV-1a (<<fnv>>) into
- a fixed-size bucket vector using the feature hashing trick (<<weinberger2009>>).
- The general model uses bigrams, trigrams, 3-char suffixes, 3-char prefixes, word
- unigrams, and CJK character unigrams; the short-text model uses bigrams, trigrams,
- 4-grams, 5-grams, and word unigrams (no suffixes/prefixes).
-3. **Classification** — multinomial logistic regression / softmax
- (<<bishop2006>>) with INT8 quantized weights (<<jacob2018>>)
+Character n-gram language identification (<<cavnar1994>>) in three stages:
+
+1. **Preprocessing** — truncate, strip URLs/emails, NFC normalize, case fold
+2. **Feature extraction** — n-grams and script features hashed via FNV-1a
+ (<<fnv>>) into a fixed-size bucket vector using the feature hashing trick
+ (<<weinberger2009>>)
+3. **Classification** — a linear multinomial model (<<bishop2006>>) with INT8
+ quantized weights (<<jacob2018>>), scored as raw logits
=== Feature Extraction
-The `ScriptAwareFeatureExtractor` (used by the general model) produces the
-following features from preprocessed text:
-
-* **Character bigrams** — adjacent character pairs with word-boundary sentinels
- (`_`). For example, `"hello"` produces `_h`, `he`, `el`, `ll`, `lo`, `o_`.
-* **Character trigrams** — overlapping character triples including boundary
- trigrams at word start (`_ab`) and end (`ab_`).
-* **3-char word suffixes** — the last three characters of each word (words of
- 3+ codepoints). Suffixes are highly discriminative for inflected languages.
-* **3-char word prefixes** — the first three characters of each word (words of
- 3+ codepoints). Complements suffixes for prefix-heavy morphological systems.
-* **Whole-word unigrams** — full word tokens hashed as features (2–30 codepoints).
- Captures function words and short words that are highly discriminative for many
- languages (e.g., "the", "de", "и").
-* **CJK character unigrams** — individual Han, Hiragana, and Katakana characters
- emitted as features. CJK scripts pack much more information per character than
- alphabetic scripts, making unigrams valuable.
-* **CJK space bridging** — when CJK characters are separated by whitespace
- (common in tokenized corpora), the extractor bridges the gap and still
- produces bigrams across the space. This prevents tokenization artifacts from
- degrading CJK language detection.
-* **Japanese script family** — Han, Hiragana, and Katakana are treated as a
- single script "family" for boundary detection. Japanese text freely mixes all
- three scripts within words and phrases, so script transitions within this
- family do not create word boundaries.
-
-All features are hashed to bucket indices via FNV-1a. The current model uses
-16,384 buckets.
+`SaltedNgramFeatureExtractor` distinguishes feature types by a one-byte salt
+prefix on the FNV hash chain rather than by sentinel characters, so an n-gram
+always contains N real characters. Word position (BOW / EOW / FULL_WORD / MID)
+is part of the salt.
+
+* **Character bigrams, trigrams, 4-grams** — every contiguous run within a word,
+ each carrying its position salt.
+* **Short complete words (1–4 chars)** — a FULL_WORD salt on the matching n-gram
+ order. This captures the function words ("the", "de", "и") that discriminate
+ many languages, without a separate word-unigram feature.
+* **Word bigrams** — a short anchor word (≤ 3 codepoints) plus the word after it,
+ e.g. "the X", "de X".
+* **CJK/kana unigrams** — individual ideographic and kana codepoints under their
+ own salt. CJK scripts pack far more information per character than alphabetic
+ scripts, and they have no word boundaries to anchor to.
+* **Script blocks** — per-script letter counts and script-transition counts,
+ giving the linear classifier explicit script signal.
+
+N-grams are not script-salted: different scripts occupy different codepoint
+ranges, so their hashes separate naturally. Feature vectors are L2-normalized.
+The shipped model uses 32,768 buckets.
=== Preprocessing Pipeline
@@ -86,10 +75,15 @@ raw text
→ NFC Unicode normalization
→ skip transparent characters (see below)
→ case fold via Character.toLowerCase()
- → extract features (bigrams, word unigrams, CJK unigrams)
+ → extract features
→ FNV-1a hash each feature into bucket vector
+ → L2 normalize
----
+The 100K truncation is `CharSoupFeatureExtractor.MAX_TEXT_LENGTH`; the rolling
+buffer cap is adjustable with `setMaxLength(int)`. `hasEnoughText()` returns
+true at 10,000 buffered characters, so a streaming caller can stop feeding there.
+
=== Transparent Character Handling
Certain codepoints are treated as _transparent_ — they are skipped entirely so that
@@ -112,130 +106,91 @@ Arabic and Hebrew processing:
A fast guard (`cp < 0x0300`) short-circuits the check for ASCII and Latin text,
adding zero overhead to the common case.
-== Models
-
-CharSoupLanguageDetector ships with two complementary models:
-
-=== General Model (`langdetect.bin`)
-
-The general model covers *203 languages* trained on Wikipedia dumps as the
-primary source, supplemented by
-https://research.google/blog/madlad-400-a-multilingual-and-document-level-large-audited-dataset/[MADLAD-400]
-for languages with insufficient Wikipedia coverage. It uses 16,384 hash buckets
-and `ScriptAwareFeatureExtractor`: character bigrams, character trigrams, 3-char
-word suffixes, 3-char word prefixes, whole-word unigrams, and CJK character unigrams.
-
-=== Short-Text Model (`langdetect-short.bin`)
-
-The short-text model is optimized for inputs under ~200 characters — document
-titles, metadata fields, subject lines, captions, and similar short strings where
-the general model loses confidence. It covers 123 carefully selected languages
-(those that generalize well at short lengths and are not excessively confusable
-with each other) and uses 32,768 hash buckets with `ResearchFeatureExtractor`
-(bigrams + trigrams + 4-grams + word unigrams). The richer n-gram features
-compensate for the reduced token count at short text lengths.
-
-=== Automatic Model Selection
-
-By default, `CharSoupLanguageDetector` selects the model automatically per chunk
-using two gates (evaluated in `AUTOMATIC` strategy mode):
-
-1. **Length gate** — if the chunk is shorter than 200 characters, use the
- short-text model.
-2. **Feature-density gate** — if the n-gram emission count from the general
- extractor is below 200, use the short-text model regardless of character
- length. This catches degenerate inputs such as a long string of whitespace
- followed by a single word, where character length alone would incorrectly
- route to the general model.
-
-If the short-text model binary is absent from the classpath, both gates fall
-back to the general model transparently.
-
-=== Overriding Model Selection
-
-The automatic gates (200-char length, 200-feature density) and the strategy
-choice (always-short / always-general / automatic) are currently fixed in
-`CharSoupLanguageDetector`. Programmatic knobs are limited to
-`setMaxLength(int)` for the rolling buffer and `setPriors(Map)` for language
-priors.
-
-`CharSoupLanguageDetector` is registered as `@TikaComponent(name =
-"charsoup-language-detector")` and implements `SelfConfiguring`, so once a
-config record is added, declarative tuning will be available via the standard
-`parse-context` JSON section:
-
-[source,json]
-----
-{
- "parse-context": {
- "charsoup-language-detector": {
- /* future tuning fields will go here */
- }
- }
-}
-----
-
-See xref:migration-to-4x/serialization-4x.adoc#discovering-friendly-names[Discovering the friendly name for a component] for how `SelfConfiguring` components are resolved at runtime.
-
-== Training the Models
-
-Training is fully reproducible from source. For step-by-step instructions,
-corpus download scripts, training commands, and detailed benchmark comparisons,
-see xref:advanced/language-detection-build.adoc[Building the Language Detector].
+== The bundled model
+
+One model covers every text length:
+`org/apache/tika/langdetect/charsoup/langdetect-20260320.bin`, loaded statically
+by `CharSoupLanguageDetector`.
+
+[cols="1,3"]
+|===
+|Languages |204 (see xref:advanced/charsoup-supported-languages.adoc[the full list])
+|Corpus |Wikipedia dumps, supplemented by
+https://huggingface.co/datasets/allenai/MADLAD-400[MADLAD-400] where Wikipedia
+coverage is thin
+|Hash buckets |32,768
+|Feature extractor |`SaltedNgramFeatureExtractor` — trigrams, 4-grams, script
+blocks, word bigrams, L2 normalization
+|Size |6.4 MB on disk. The file is uncompressed and the weight matrix is retained
+as the single INT8 `byte[32768 x 204]` it is stored as, so the retained model is
+about the same size; the ~8.1 MB quoted in the benchmark logs is a measured
+used-heap delta across the load call, which includes transient buffers.
+|===
+
+The model file records the feature flags it was trained with, and the detector
+refuses to load if the extractor's flags disagree — a mismatch would produce
+silently wrong scores rather than an error.
+
+=== Scoring
+
+Inference works on raw logits; no softmax distribution is computed. Confidence is
+the *margin* between the top two logits after confusable groups are collapsed:
+`sigmoid(top − second)`. That is invariant to the number of classes, so the same
+threshold is meaningful for a short snippet and a full document. Per-class
+`rawScore` is `sigmoid(logit_c − best_competitor_logit)`, so the winner is above
+0.5 and everything else below.
+
+=== Tuning
+
+`CharSoupLanguageDetector` is registered as
+`@TikaComponent(name = "charsoup-language-detector")` and implements
+`SelfConfiguring`, so it resolves by friendly name in the `parse-context` JSON
+section — see
+xref:migration-to-4x/serialization-4x.adoc#discovering-friendly-names[Discovering the friendly name for a component].
+It exposes no config record yet; the programmatic knobs are `setMaxLength(int)`
+for the rolling buffer and `setPriors(Map)` for language priors.
+
+== Training the model
+
+Training is reproducible from source. For corpus preparation, training commands,
+and benchmark comparisons, see
+xref:advanced/language-detection-build.adoc[Building the Language Detector].
== Model Format (LDM1)
-The binary model format is:
-
[source]
----
-4 bytes magic: 0x4C444D31 ("LDM1")
-4 bytes numBuckets (int32 big-endian)
-4 bytes numClasses (int32 big-endian)
-
-For each class:
- 2 bytes label length (uint16)
- N bytes label (UTF-8)
-
-numClasses × 4 bytes per-class scales (float32)
-numClasses × 4 bytes per-class biases (float32)
-numBuckets × numClasses bytes weight matrix (int8, bucket-major)
+int32 magic 0x4C444D31 ("LDM1")
+int32 version (1, or 2 with feature flags)
+int32 numBuckets
+int32 numClasses
+int32 featureFlags (version 2 only)
+
+for each class:
+ uint16 label length
+ N bytes label (UTF-8)
+
+float32[numClasses] per-class scales
+float32[numClasses] per-class biases
+int8[numBuckets × numClasses] weight matrix, bucket-major
----
-The weight matrix is stored in **bucket-major** order: for each bucket, all class
-weights are contiguous. This layout is optimal for sparse inference, where only
-non-zero buckets are visited.
-
-The general model is stored at
-`org/apache/tika/langdetect/charsoup/langdetect.bin` and the short-text model
-at `org/apache/tika/langdetect/charsoup/langdetect-short.bin`. Both are loaded
-statically by `CharSoupLanguageDetector`; the short-text model load is gracefully
-skipped if the resource is absent.
-
-=== Memory-Mapped Loading
-
-For deployment scenarios that benefit from off-heap memory (e.g., multiple JVM
-instances sharing the same model), the `CharSoupModel.loadMapped(Path)` method
-loads the model via `MappedByteBuffer`. A companion `saveSplit(Path, Path)`
-method writes the raw weights and metadata as separate files for true zero-copy
-loading.
-
-For the default classpath resources (general model ~3.2 MB, short-text model
-~3.8 MB), heap loading is used and the performance difference is negligible.
+The weight matrix is **bucket-major**: all class weights for a bucket are
+contiguous. Inference visits only non-zero buckets, so this is the layout that
+keeps that walk sequential.
== WordTokenizer (tika-eval integration)
-The same preprocessing pipeline is exposed as a general-purpose word tokenizer
-via `org.apache.tika.langdetect.charsoup.core.WordTokenizer`. This replaces the former
-Lucene-based tokenizer in `tika-eval`:
+The same preprocessing pipeline is exposed as a general-purpose word tokenizer,
+`org.apache.tika.langdetect.charsoup.core.WordTokenizer`, which is what
+`tika-eval` tokenizes with:
* `tokenize(String)` — alphabetic and ideographic tokens only (CJK bigrams)
* `tokenizeAlphanumeric(String, Consumer)` — also emits digit-only runs as tokens
-The alphanumeric variant is used by `tika-eval` so it can still distinguish
-alphabetic token count from total (alphanumeric) token count. The alpha-only
-variant is a separate code path with zero per-character overhead from the
-numeric check, keeping the language detection hot path fast.
+`tika-eval` needs the alphanumeric variant to distinguish alphabetic token count
+from total token count. The alpha-only variant is a separate code path so the
+language-detection hot path pays nothing for the numeric check.
== References
@@ -289,15 +244,6 @@ The language detector draws on several well-established techniques.
Standard reference for multinomial logistic regression (softmax
classification), the model used for the final prediction layer.
-- [[[goldhahn2012]]] D. Goldhahn, T. Eckart, and U. Quasthoff,
- "Building Large Monolingual Dictionaries at the Leipzig Corpora Collection:
- From 100 to 200 Languages,"
- in _Proceedings of the Eighth International Conference on Language Resources
- and Evaluation (LREC)_, Istanbul, Turkey, 2012, pp. 759–765. +
- The Leipzig Corpora Collection was used in early model versions (v1/v2).
- Current models (v7+) use Wikipedia dumps as the primary corpus. +
- https://aclanthology.org/L12-1154/
-
- [[[jacob2018]]] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko,
"Quantization and Training of Neural Networks for Efficient
Integer-Arithmetic-Only Inference,"
diff --git a/docs/modules/ROOT/pages/advanced/local-vlm-server.adoc b/docs/modules/ROOT/pages/advanced/local-vlm-server.adoc
index 0391ff9da6..d0a0ab2271 100644
--- a/docs/modules/ROOT/pages/advanced/local-vlm-server.adoc
+++ b/docs/modules/ROOT/pages/advanced/local-vlm-server.adoc
@@ -30,7 +30,9 @@ This is useful for:
* Evaluation and testing
The server wraps a Hugging Face model in a lightweight FastAPI app that
-exposes the `/v1/chat/completions` endpoint in the OpenAI format.
+exposes the `/v1/chat/completions` and `/v1/models` endpoints in the OpenAI
+format. `OpenAIVLMParser` probes `/v1/models` at startup and stays disabled
+for the whole run if that probe fails, so the server must answer it.
== Prerequisites
@@ -161,6 +163,7 @@ class ChatResponse(BaseModel):
app = FastAPI(title="VLM server")
model = None
+model_name = None
processor = None
device = None
@@ -205,6 +208,12 @@ def health():
return {"status": "ok"}
+# Tika's OpenAIVLMParser health-checks this at init; a 404 here disables the parser.
+@app.get("/v1/models")
+def list_models():
+ return {"object": "list", "data": [{"id": model_name, "object": "model"}]}
+
+
@app.post("/v1/chat/completions")
def chat_completions(req: ChatRequest):
conversation, images = parse_messages(req.messages)
@@ -262,7 +271,7 @@ def chat_completions(req: ChatRequest):
def main():
- global model, processor, device
+ global model, model_name, processor, device
parser = argparse.ArgumentParser(
description="VLM OpenAI-compatible server (localhost only)"
@@ -275,6 +284,7 @@ def main():
"--port", type=int, default=8000, help="Port (default: 8000)"
)
args = parser.parse_args()
+ model_name = args.model
if torch.backends.mps.is_available():
device = "mps"
@@ -321,6 +331,9 @@ network.
----
curl http://localhost:8000/health
# {"status":"ok"}
+
+curl http://localhost:8000/v1/models
+# {"object":"list","data":[{"id":"jinaai/jina-vlm","object":"model"}]}
----
== Configure Tika to use it
diff --git a/docs/modules/ROOT/pages/advanced/robustness.adoc b/docs/modules/ROOT/pages/advanced/robustness.adoc
index c6ad52948c..6dbc019786 100644
--- a/docs/modules/ROOT/pages/advanced/robustness.adoc
+++ b/docs/modules/ROOT/pages/advanced/robustness.adoc
@@ -24,114 +24,64 @@ When processing documents at scale, you must implement protective measures.
IMPORTANT: Avoid running Tika in the same process as critical infrastructure like
indexers or search systems.
-== Process Isolation
+== Process isolation
-The primary defense against parser failures is process isolation. By running parsers
-in separate processes, you protect your main application from:
+Process isolation is the primary defense: a parser that OOMs, loops forever,
+crashes native code, or exhausts a file-descriptor pool takes down a forked JVM
+(a *fork*), not your application.
-* OutOfMemoryErrors
-* Infinite loops
-* Native code crashes
-* Resource exhaustion
+xref:pipes/index.adoc[Tika Pipes] is the single forking mechanism: everything
+that parses in a separate process goes through it.
-=== Tika 4.x
-
-**In Tika 4.x, xref:pipes/index.adoc[Tika Pipes] is the recommended approach for
-robust document processing.** It provides:
-
-* Automatic process isolation
-* Fault tolerance and recovery
-* Scalable parallel processing
-* Unified architecture for all deployment scenarios
-
-Pipes can be used in multiple ways:
-
-* **Programmatically** - Via `PipesForkParser` in the `tika-pipes-fork-parser` module
- (see xref:using-tika/java-api/getting-started.adoc[Java API Getting Started])
-* **Via tika-server** - REST endpoints for pipes-based processing
-* **Via tika-grpc** - gRPC interface with pipes backend
-
-In Tika 4.x, the approach to robustness has been simplified. Previous versions offered
-four different forking mechanisms:
-
-[cols="1,2,1"]
+[cols="1,3"]
|===
-|Mechanism |Description |Status in 4.x
+|Entry point |How it forks
-|ForkParser
-|Spawned child processes for individual parse operations
-|Deprecated
+|`PipesForkParser`
+|Programmatic. `tika-pipes/tika-pipes-fork-parser`; see
+xref:using-tika/java-api/getting-started.adoc[Java API Getting Started].
-|tika-batch
-|Desktop/VM-scale batch processing
-|Deprecated
+|tika-app
+|`-f` / `--fork` runs a single file in a fork; `-i`/`-o` batch mode runs a pool
+of forks.
-|tika-server (forked mode)
-|REST server with forked parsing processes
-|Available, but Pipes recommended
+|tika-server
+|Every parse request is served by a fork. Not optional. Clients must
+handle fork restarts gracefully. The `/async` and `/pipes` endpoints
+additionally require `allowPipes=true`.
-|tika-pipes
-|Scalable, fault-tolerant pipeline processing
-|*Recommended approach*
+|tika-grpc
+|gRPC interface over the same Pipes backend.
|===
-=== Tika 3.x and Earlier
-
-If you are using Tika 3.x or earlier, you have several options for process isolation:
-
-ForkParser::
-Spawns child processes to protect against out-of-memory errors and infinite loops.
-Suitable for programmatic use in Java applications.
-
-tika-batch::
-For desktop/VM-scale processing (not cloud-scale):
-+
-[source,bash]
-----
-java -jar tika-app.jar -i <input_dir> -o <output_dir>
-----
-
-tika-server::
-In version 2.x and later, parsing defaults to forked processes. Clients must handle
-tika-server restarts gracefully.
-
-tika-pipes::
-Available through programmatic use, tika-app `-a` option, or tika-server's `/async`
-and `/pipes` endpoints (the server endpoints require `allowPipes=true`).
-
-== Security Testing and Prevention
-
-The Apache Tika team implements several measures to identify and prevent vulnerabilities:
-
-* **Regression testing** against ~2 million files from Common Crawl before releases
-* **Code reviews** of dependencies to identify vulnerability patterns
-* **Fuzzing modules** for automated vulnerability discovery
-* **Collaboration** with security researchers
-* **Maintained forks** of parsers with critical fixes (released independently when needed)
-* **Public documentation** of vulnerabilities at xref:security.adoc[security page]
-
-== MockParser for Testing
+== What the project does upstream
-Tika provides a `MockParser` tool for testing your system's robustness. You can
-configure it to simulate various failure modes:
+* Regression testing against ~2 million Common Crawl files before releases
+* Dependency code review for vulnerability patterns
+* Fuzzing modules for automated vulnerability discovery
+* Collaboration with security researchers
+* Maintained source forks of upstream parser libraries carrying critical fixes,
+ released independently when needed
+* Public vulnerability disclosure on the xref:security.adoc[security page]
-* Infinite loops
-* OutOfMemoryErrors
-* Excessive runtime
-* Large output generation
+== Testing your integration with MockParser
-This allows you to verify that your integration handles parser failures gracefully.
+`MockParser` (tika-core test jar) simulates infinite loops, OutOfMemoryErrors,
+excessive runtime, and huge output on demand, so you can verify that your
+integration survives each failure mode rather than hoping it does.
== Recommendations
-1. **Use Tika Pipes** (4.x) for production workloads with untrusted content
-2. **Isolate Tika** from critical systems - never run in the same JVM as your indexer
-3. **Set timeouts** for all parsing operations
-4. **Monitor memory usage** and set appropriate limits
-5. **Plan for failures** - your system should handle parser crashes gracefully
-6. **Stay updated** - apply security updates promptly
+. Never run Tika in the same JVM as an indexer, search system, or anything else
+ whose availability matters.
+. Set timeouts and xref:advanced/setting-limits.adoc[limits] on every parse of
+ untrusted content.
+. Expect fork crashes and restarts; design the client to retry or record and
+ move on.
+. Apply security updates promptly.
-== Further Reading
+== Further reading
-* xref:pipes/index.adoc[Tika Pipes] - Recommended approach for robust processing
-* xref:security.adoc[Security] - Known vulnerabilities and security model
+* xref:pipes/index.adoc[Tika Pipes] — the forking runtime
+* xref:advanced/setting-limits.adoc[Setting Limits] — resource bounds
+* xref:security.adoc[Security] — known vulnerabilities and the security model
diff --git a/docs/modules/ROOT/pages/advanced/setting-limits.adoc b/docs/modules/ROOT/pages/advanced/setting-limits.adoc
index d05a2caa55..2d933a3a6b 100644
--- a/docs/modules/ROOT/pages/advanced/setting-limits.adoc
+++ b/docs/modules/ROOT/pages/advanced/setting-limits.adoc
@@ -17,20 +17,42 @@
= Setting Limits
-When processing untrusted documents, it's important to set limits on resource consumption
-to prevent denial-of-service attacks and protect against malicious or pathological files.
-Tika provides several mechanisms for limiting resource usage during parsing.
+Untrusted documents can be pathological: deeply nested, self-expanding, or
+crafted to run forever. Every limit below is configured in the `parse-context`
+section of the JSON config, loaded into the `ParseContext`, and enforced
+throughout the parse.
-== Overview
+== Limits at a glance
-Tika 4.x provides a unified configuration system for all limits through the `parse-context`
-section of the JSON configuration file. All limits are loaded into the `ParseContext` and
-flow through the parsing pipeline.
+[cols="2,2,4"]
+|===
+|`parse-context` key |Class |Bounds
+
+|`embedded-limits`
+|`EmbeddedLimits`
+|Depth and count of embedded documents. Unlimited by default.
+
+|`output-limits`
+|`OutputLimits`
+|Extracted characters, XML nesting, package nesting, zip-bomb ratio.
+
+|`timeout-limits`
+|`TimeoutLimits`
+|Total wall-clock budget per task and the stall detector.
+
+|`unpack-config`
+|`UnpackConfig`
+|Total bytes written by `ParseMode.UNPACK`. Pipes only.
+
+|`standard-metadata-limiter-factory`
+|`StandardMetadataLimiterFactory`
+|Metadata size, per-field size, key size, values per field, field allow/deny lists.
+|===
-=== Complete Example
+== Configuring limits
-Here's a comprehensive example showing all limit configurations together.
-This is the same configuration tested in `AllLimitsTest.java`:
+Every limit group is a `parse-context` block. This is the configuration
+exercised by `AllLimitsTest` (`tika-serialization/src/test/resources/configs/all-limits-test.json`):
[source,json]
----
@@ -65,32 +87,34 @@ This is the same configuration tested in `AllLimitsTest.java`:
}
----
-Configuration file: `tika-serialization/src/test/resources/configs/all-limits-test.json`
-
-=== Loading Limits
-
-Use `TikaLoader.loadParseContext()` to load all configured limits into a `ParseContext`:
+`TikaLoader.loadParseContext()` loads them all; each class has a static
+`get(ParseContext)` that returns the configured instance or defaults:
[source,java]
----
-TikaLoader loader = TikaLoader.load(configPath);
-ParseContext context = loader.loadParseContext();
+ParseContext context = TikaLoader.load(configPath).loadParseContext();
-// Limits are now available from the context
-EmbeddedLimits embeddedLimits = context.get(EmbeddedLimits.class);
-OutputLimits outputLimits = context.get(OutputLimits.class);
-TimeoutLimits timeoutLimits = context.get(TimeoutLimits.class);
+EmbeddedLimits embedded = EmbeddedLimits.get(context);
+OutputLimits output = OutputLimits.get(context);
+TimeoutLimits timeouts = TimeoutLimits.get(context);
----
-See test: `tika-serialization/src/test/java/org/apache/tika/config/AllLimitsTest.java`
+To set them programmatically, construct and put them on the context — the same
+pattern for every limit class:
-== Embedded Document Limits
+[source,java]
+----
+context.set(EmbeddedLimits.class, new EmbeddedLimits(10, true, 500, false));
+context.set(OutputLimits.class, new OutputLimits(50000, true, 50, 5, 500000, 50));
+context.set(TimeoutLimits.class, new TimeoutLimits(7200000, 120000));
+----
+
+Tests: `AllLimitsTest`, `EmbeddedLimitsTest`, `OutputLimitsTest`, and
+`TimeoutLimitsTest` under `tika-serialization/src/test/java/org/apache/tika/config/`.
-The `EmbeddedLimits` class controls how deeply nested and how many embedded documents
-are processed. This is critical for protecting against "zip bomb" style attacks where
-documents contain deeply nested or numerous embedded files.
+== Embedded document limits
-=== Configuration Options
+`EmbeddedLimits` bounds how deep and how many embedded documents are parsed.
[cols="2,1,3"]
|===
@@ -98,87 +122,39 @@ documents contain deeply nested or numerous embedded files.
|`maxDepth`
|-1 (unlimited)
-|Maximum nesting depth for embedded documents. When reached, recursion stops but
-siblings at the current level continue to be processed.
+|Maximum nesting depth. Recursion stops at the limit; siblings at the current
+level still parse.
|`throwOnMaxDepth`
|false
-|Whether to throw an `EmbeddedLimitReachedException` when maxDepth is reached.
-If false, processing continues and `tk:exception:embedded-depth-limit-reached=true` is set in metadata.
+|Throw `EmbeddedLimitReachedException` at `maxDepth` instead of continuing and
+setting `tk:exception:embedded-depth-limit-reached=true`.
|`maxCount`
|-1 (unlimited)
-|Maximum total number of embedded documents to process. When reached, processing
-stops immediately.
+|Maximum total embedded documents. Processing stops immediately when reached.
|`throwOnMaxCount`
|false
-|Whether to throw an `EmbeddedLimitReachedException` when maxCount is reached.
-If false, processing continues and `tk:exception:embedded-resource-limit-reached=true` is set.
+|Throw `EmbeddedLimitReachedException` at `maxCount` instead of continuing and
+setting `tk:exception:embedded-resource-limit-reached=true`.
|===
-=== maxDepth Behavior
-
-When the depth limit is reached, recursion stops but siblings at the current level
-continue to be processed. For example, with `maxDepth=1`:
+With `maxDepth=1`, depth-1 siblings all parse and their children do not:
[source]
----
container.zip (depth 0)
-├── doc1.docx (depth 1) ✓ PARSED
-│ ├── image1.png (depth 2) ✗ NOT PARSED (exceeds maxDepth)
-│ └── embed.xlsx (depth 2) ✗ NOT PARSED (exceeds maxDepth)
-├── doc2.pdf (depth 1) ✓ PARSED (sibling at same level)
-└── doc3.txt (depth 1) ✓ PARSED (sibling at same level)
+├── doc1.docx (depth 1) PARSED
+│ ├── image1.png (depth 2) NOT PARSED
+│ └── embed.xlsx (depth 2) NOT PARSED
+├── doc2.pdf (depth 1) PARSED
+└── doc3.txt (depth 1) PARSED
----
-=== JSON Configuration
+== Output limits
-[source,json]
-----
-{
- "parse-context": {
- "embedded-limits": {
- "maxDepth": 5,
- "throwOnMaxDepth": true,
- "maxCount": 100,
- "throwOnMaxCount": false
- }
- }
-}
-----
-
-Configuration file: `tika-serialization/src/test/resources/configs/embedded-limits-test.json`
-
-=== Java API
-
-[source,java]
-----
-// Create with constructor
-EmbeddedLimits limits = new EmbeddedLimits(10, true, 500, false);
-
-// Or use setters
-EmbeddedLimits limits = new EmbeddedLimits();
-limits.setMaxDepth(10);
-limits.setThrowOnMaxDepth(true);
-limits.setMaxCount(500);
-limits.setThrowOnMaxCount(false);
-
-// Add to ParseContext
-context.set(EmbeddedLimits.class, limits);
-
-// Helper method to get limits with defaults
-EmbeddedLimits limits = EmbeddedLimits.get(context); // Returns defaults if not set
-----
-
-See test: `tika-serialization/src/test/java/org/apache/tika/config/EmbeddedLimitsTest.java`
-
-== Output Limits
-
-The `OutputLimits` class controls limits on parsing output including text extraction
-and protection against zip bombs.
-
-=== Configuration Options
+`OutputLimits` bounds extracted text and structural expansion.
[cols="2,1,3"]
|===
@@ -186,329 +162,203 @@ and protection against zip bombs.
|`writeLimit`
|-1 (unlimited)
-|Maximum characters of text to extract. When reached, extraction stops.
+|Maximum characters of text to extract. Extraction stops when reached.
|`throwOnWriteLimit`
|false
-|Whether to throw a `WriteLimitReachedException` when writeLimit is reached.
+|Throw `WriteLimitReachedException` at `writeLimit` instead of stopping and setting
+`tk:exception:write-limit-reached=true`.
|`maxXmlDepth`
|100
-|Maximum XML element nesting depth. Protects against XML bomb attacks.
+|Maximum XML element nesting depth. Guards against XML bombs.
|`maxPackageEntryDepth`
|10
-|Maximum depth of nested package entries (e.g., zip within zip).
+|Maximum depth of nested package entries (zip within zip).
|`zipBombThreshold`
|1,000,000
-|Minimum decompressed size (in bytes) before zip bomb detection activates.
+|Extracted characters before the zip-bomb ratio check activates.
|`zipBombRatio`
|100
-|Maximum ratio of decompressed to compressed size before flagging as zip bomb.
+|Maximum ratio of extracted characters to input bytes read before flagging a zip bomb.
|===
-=== JSON Configuration
-
-[source,json]
-----
-{
- "parse-context": {
- "output-limits": {
- "writeLimit": 50000,
- "throwOnWriteLimit": true,
- "maxXmlDepth": 50,
- "maxPackageEntryDepth": 5,
- "zipBombThreshold": 500000,
- "zipBombRatio": 50
- }
- }
-}
-----
-
-Configuration file: `tika-serialization/src/test/resources/configs/output-limits-test.json`
-
-=== Java API
-
-[source,java]
-----
-OutputLimits limits = new OutputLimits(50000, true, 50, 5, 500000, 50);
-context.set(OutputLimits.class, limits);
-
-// Helper method
-OutputLimits limits = OutputLimits.get(context);
-----
-
-See test: `tika-serialization/src/test/java/org/apache/tika/config/OutputLimitsTest.java`
-
-== Timeout Limits
-
-The `TimeoutLimits` class controls time-based limits for parsing operations. Tika 4.x
-uses a *two-tier* timeout: one bound on total wall-clock time, and one on time elapsed
-since the parser last reported progress.
+== Timeout limits
-=== Configuration Options
+`TimeoutLimits` applies two independent bounds: one on total wall-clock time,
+one on time since the parser last reported progress.
[cols="2,1,3"]
|===
|Setting |Default |Description
|`totalTaskTimeoutMillis`
-|3600000 (1 hour)
-|Maximum wall-clock time in milliseconds for the entire parse task, including all of
-its embedded documents. Every per-parser timeout is clipped to whatever remains of this
-budget, no matter how it is itself configured.
+|3,600,000 (1 hour)
+|Wall-clock budget for the whole task, embedded documents included. Every
+per-parser timeout is clipped to what remains of this budget, however it is
+itself configured.
|`progressTimeoutMillis`
-|120000 (2 minutes)
-|Maximum time in milliseconds the task may go silent before it is considered hung.
-Catches infinite loops and hung processes. Enforced (the task actually killed) only
-when the parse runs in a forked JVM -- Pipes mode or tika-app `--fork`; in plain
-in-process library use nothing kills the parsing thread.
+|120,000 (2 minutes)
+|How long the task may go silent before it counts as hung. Enforced — the task
+actually killed — only when the parse runs in a forked JVM (Pipes, or tika-app
+`--fork`); in-process library use has nothing to kill the parsing thread.
|`throwOnDeadline`
|false
-|Whether a `totalTaskTimeoutMillis` exhaustion should throw `EmbeddedLimitReachedException`
-instead of skipping remaining embedded documents cleanly and reporting a
-`PARTIAL_TIMEOUT` result. See xref:pipes/timeouts.adoc#_throwondeadline_failing_hard_instead_of_partial[throwOnDeadline]
-for the pipes-mode caveat before enabling this.
+|Throw `EmbeddedLimitReachedException` on `totalTaskTimeoutMillis` exhaustion
+instead of skipping the remaining embedded documents and reporting
+`PARTIAL_TIMEOUT`. See
+xref:pipes/timeouts.adoc#_throwondeadline_failing_hard_instead_of_partial[throwOnDeadline]
+for the pipes-mode caveat before enabling it.
|===
[IMPORTANT]
====
-The two bounds are independent, not tiered: `progressTimeoutMillis` is a *stall
-detector*, not a ceiling on any single operation's runtime. A per-parser timeout (e.g.
-`tesseract-ocr-parser.timeoutMillis`) is honored up to `min(configured, time remaining
-in totalTaskTimeoutMillis)` — `progressTimeoutMillis` never enters that calculation.
-
-While Tika waits on a bounded external call (an OCR process, an external parser), the
-wait itself checkpoints progress periodically, so a single legitimate call may run far
-longer than `progressTimeoutMillis` without tripping the stall detector — a 10-minute
-`readpst` invocation does not need `progressTimeoutMillis` raised to 10 minutes. The
-stall detector only fires on genuine silence: an in-JVM hang, or a wedged process that
-even checkpointing cannot report past.
-
-Budgets compose recursively regardless of nesting: a PDF inside a zip inside an email
-draws every operation, at any depth, from the same `totalTaskTimeoutMillis` budget for
-that top-level task.
-
-For the full mental model, worked scenarios, and how each outcome is reported (metadata
-flags, result status, exception messages), see xref:pipes/timeouts.adoc[Timeouts].
+`progressTimeoutMillis` is a *stall detector*, not a ceiling on any single
+operation. A per-parser timeout (e.g. `tesseract-ocr-parser.timeoutMillis`) is
+honored up to `min(configured, time remaining in totalTaskTimeoutMillis)`;
+`progressTimeoutMillis` never enters that calculation.
+
+Waits on a bounded external call checkpoint progress periodically, so a
+legitimate 10-minute `readpst` invocation does not need `progressTimeoutMillis`
+raised to 10 minutes. The stall detector fires only on genuine silence: an
+in-JVM hang, or a process so wedged that checkpointing cannot report past it.
+
+Budgets compose recursively — a PDF inside a zip inside an email draws from the
+one `totalTaskTimeoutMillis` budget for the top-level task. For worked
+scenarios and how each outcome is reported, see xref:pipes/timeouts.adoc[Timeouts].
====
-=== JSON Configuration
-
-[source,json]
-----
-{
- "parse-context": {
- "timeout-limits": {
- "totalTaskTimeoutMillis": 7200000,
- "progressTimeoutMillis": 120000
- }
- }
-}
-----
+== Embedded byte extraction limits
-Configuration file: `tika-serialization/src/test/resources/configs/timeout-limits-test.json`
-
-=== Java API
-
-[source,java]
-----
-TimeoutLimits limits = new TimeoutLimits(7200000, 120000);
-limits.setThrowOnDeadline(false);
-context.set(TimeoutLimits.class, limits);
-
-// Helper method
-TimeoutLimits limits = TimeoutLimits.get(context);
-----
-
-See test: `tika-serialization/src/test/java/org/apache/tika/config/TimeoutLimitsTest.java`
-
-== Embedded Byte Extraction Limits
-
-When extracting embedded document bytes using `ParseMode.UNPACK`, the `UnpackConfig` class
-provides safety limits on total bytes extracted. This protects against zip bombs and other
-malicious files that may expand to enormous sizes when unpacked.
-
-=== Configuration Options
+`ParseMode.UNPACK` writes embedded bytes out; `UnpackConfig.maxUnpackBytes`
+caps the total.
[cols="2,1,3"]
|===
|Setting |Default |Description
|`maxUnpackBytes`
-|10 GB
-|Maximum total bytes to extract from all embedded documents per file. Set to -1 for
-unlimited (not recommended for untrusted input). 0 is not unlimited -- it means zero
-bytes.
+|10 GiB
+|Maximum total bytes extracted from all embedded documents per file. `-1` is
+unlimited (not advisable for untrusted input); `0` means zero bytes, not unlimited.
|===
-=== Behavior
-
-When the byte limit is reached:
-
-* Extraction stops for remaining embedded documents
-* An exception is logged but processing continues
-* Already-extracted bytes are kept
-* The parse result status is `PARSE_SUCCESS_WITH_EXCEPTION`
-
-=== JSON Configuration
+At the limit, extraction stops for the remaining embedded documents and
+already-extracted bytes are kept. The parse still reports success: the truncation
+shows up only as a WARN in the fork's log, not in the metadata or the result status.
[source,json]
----
{
- "pipes": {
- "parseMode": "UNPACK"
- },
+ "pipes": { "parseMode": "UNPACK" },
"parse-context": {
- "unpack-config": {
- "maxUnpackBytes": 104857600
- }
+ "unpack-config": { "maxUnpackBytes": 104857600 }
}
}
----
-This limits extraction to 100 MB total.
-
-=== Java API
-
-[source,java]
-----
-UnpackConfig config = new UnpackConfig();
-config.setMaxUnpackBytes(100 * 1024 * 1024); // 100 MB
-config.setEmitter("my-emitter");
-parseContext.set(UnpackConfig.class, config);
-parseContext.set(ParseMode.class, ParseMode.UNPACK);
-----
-
-For more details on embedded byte extraction configuration, see xref:pipes/unpack-config.adoc[Extracting Embedded Bytes].
+See xref:pipes/unpack-config.adoc[Extracting Embedded Bytes] for the rest of
+`UnpackConfig`, and `UnpackModeTest` in
+`tika-pipes/tika-pipes-integration-tests`.
-See tests: `tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java`
+== Metadata limits
-== Metadata Limits
-
-The `MetadataWriteLimiter` system allows you to constrain metadata size at write time,
-ensuring parsers cannot exceed your configured limits.
-
-=== How It Works
-
-When you configure a `MetadataWriteLimiterFactory` in the `ParseContext`, calling
-`Metadata.newInstance(parseContext)` creates a `Metadata` object with limits already applied.
-All subsequent writes to that metadata object are filtered through the limiter.
+Configuring a `MetadataWriteLimiterFactory` in the `ParseContext` makes
+`Metadata.newInstance(parseContext)` return a `Metadata` with limits already
+applied, so every subsequent write is filtered.
[source,java]
----
-// Configure the factory
StandardMetadataLimiterFactory factory = new StandardMetadataLimiterFactory();
-factory.setMaxTotalBytes(1024 * 1024); // 1 MB total
-factory.setMaxFieldSize(100 * 1024); // 100 KB per field
-factory.setMaxValuesPerField(100); // Max 100 values per multi-valued field
+factory.setMaxTotalBytes(1024 * 1024);
+factory.setMaxFieldSize(100 * 1024);
+factory.setMaxValuesPerField(100);
-// Add to ParseContext
ParseContext context = new ParseContext();
context.set(MetadataWriteLimiterFactory.class, factory);
-
-// Create limited metadata - limits are enforced from the start
Metadata metadata = Metadata.newInstance(context);
----
-=== Configuration Options
-
[cols="2,1,3"]
|===
|Setting |Default |Description
|`maxTotalBytes`
|10 MB
-|Maximum total estimated size of all metadata in UTF-16 bytes. When exceeded,
-additional metadata is silently dropped and `tk:warn:truncated-metadata` is set.
+|Total estimated size of all metadata in UTF-16 bytes. Further metadata is
+dropped and `tk:warn:truncated-metadata` is set.
|`maxFieldSize`
|100 KB
-|Maximum size of any single field's value(s) in UTF-16 bytes. Values exceeding
-this limit are truncated.
+|Maximum size of a single field's value(s) in UTF-16 bytes. Longer values are
+truncated.
|`maxKeySize`
|1024
-|Maximum length of metadata key names in UTF-16 bytes. Keys exceeding this limit
-are truncated.
+|Maximum metadata key length in UTF-16 bytes. Longer keys are truncated.
|`maxValuesPerField`
|10
-|Maximum number of values for multi-valued fields. Additional values are dropped.
+|Maximum values on a multi-valued field. Extra values are dropped.
|`includeFields`
|empty (all)
-|If non-empty, only these fields are stored (plus system fields like Content-Type).
-Use this to extract only the metadata you need.
+|If non-empty, only these fields are stored, plus the always-included fields below.
|`excludeFields`
|empty (none)
-|These fields are never stored, regardless of other settings.
+|Never stored, unless always-included.
|`includeEmpty`
|false
|Whether to store empty or null values.
|===
-=== JSON Configuration
-
-[source,json]
-----
-{
- "parsers": ["default-parser"],
- "parse-context": {
- "standard-metadata-limiter-factory": {
- "maxTotalBytes": 1048576,
- "maxFieldSize": 102400,
- "maxKeySize": 1024,
- "maxValuesPerField": 100,
- "includeFields": ["dc:title", "dc:creator", "dc:subject"],
- "excludeFields": ["pdf:unmapped-unicode-chars-per-page"]
- }
- }
-}
-----
+=== Always-included fields
-=== Always-Included Fields
+`StandardMetadataLimiter.ALWAYS_SET_FIELDS` and `ALWAYS_ADD_FIELDS` bypass
+`includeFields`/`excludeFields` and the total-size budget, because parser
+dispatch and error reporting depend on them. Everything except `tk:content` is
+still truncated at `max(maxFieldSize, 300)`.
-Certain fields are critical for Tika's operation and are always allowed, regardless
-of `includeFields` or size limits:
+Always set: `Content-Type`, `Content-Length`, `Content-Encoding`,
+`Content-Disposition`, `tk:content-type-override`,
+`tk:content-type-parser-override`, `tk:content-type-hint`, `tk:content`,
+`tk:resource-name`, `tk:exception:container-exception`,
+`access-permission:extract-content`,
+`access-permission:extract-for-accessibility`.
-* `Content-Type` - Required for parser selection
-* `Content-Length`, `Content-Encoding`, `Content-Disposition`
-* `tk:content` - The extracted text content
-* `tk:parsed-by` - Parser chain information
-* `tk:warn:*` - Warning metadata
-* Access permission fields
+Always added (multi-valued): `tk:parsed-by`, `tk:exception:embedded-exception`.
-=== Detecting Truncation
+=== Detecting truncation
-When metadata is truncated due to limits, Tika sets the metadata field
-`tk:warn:truncated-metadata` to `true`. You can check for this in your code:
+Any drop or truncation sets `tk:warn:truncated-metadata`:
[source,java]
----
if ("true".equals(metadata.get(TikaCoreProperties.TRUNCATED_METADATA))) {
- // Some metadata was dropped or truncated
log.warn("Metadata was truncated for: " + resourceName);
}
----
== Recommendations
-1. **Always set limits** when processing untrusted content
-2. **Use `includeFields`** to capture only the metadata you need
-3. **Monitor for truncation** by checking `tk:warn:truncated-metadata`
-4. **Combine with process isolation** - limits protect against memory issues,
- but xref:advanced/robustness.adoc[process isolation] protects against crashes
-5. **Test with adversarial files** - use Tika's `MockParser` to simulate extreme cases
+. Set limits whenever the content is untrusted.
+. Use `includeFields` to capture only the metadata you need.
+. Check `tk:warn:truncated-metadata` rather than guessing.
+. Combine with process isolation — limits protect against memory blowups,
+ xref:advanced/robustness.adoc[process isolation] protects against crashes.
+. Test with adversarial files; `MockParser` (tika-core test jar) simulates
+ hangs, OOMs, and huge output.
-== See Also
+== See also
-* xref:advanced/robustness.adoc[Robustness] - Process isolation and fault tolerance
-* xref:configuration/index.adoc[Configuration] - General Tika configuration
-* xref:pipes/unpack-config.adoc[Extracting Embedded Bytes] - UnpackConfig for byte extraction
+* xref:advanced/robustness.adoc[Robustness] — process isolation and fault tolerance
+* xref:configuration/index.adoc[Configuration] — general Tika configuration
+* xref:pipes/unpack-config.adoc[Extracting Embedded Bytes] — the rest of `UnpackConfig`
+* xref:pipes/timeouts.adoc[Timeouts] — the full timeout model
diff --git a/docs/modules/ROOT/pages/advanced/spooling.adoc b/docs/modules/ROOT/pages/advanced/spooling.adoc
index 29b30bf297..630706eda4 100644
--- a/docs/modules/ROOT/pages/advanced/spooling.adoc
+++ b/docs/modules/ROOT/pages/advanced/spooling.adoc
@@ -49,22 +49,22 @@ Several file formats are most efficiently processed with random access vs stream
The current architecture follows a simple principle: **each component that needs random
access is responsible for obtaining it**.
-When a detector or parser needs random access, it calls:
+A detector or parser is handed a `TikaInputStream`; when it needs random access
+it simply asks for a file:
[source,java]
----
-Path path = TikaInputStream.get(inputStream).getPath();
+Path path = tis.getPath();
// or
-File file = TikaInputStream.get(inputStream).getFile();
+File file = tis.getFile();
----
`TikaInputStream` handles the spooling transparently based on how it was initialized:
* **Initialized with `Path`**: The file is used directly for random access. No spooling needed.
* **Initialized with `byte[]`**: The bytes are kept in memory. Spooling only on demand.
-* **Initialized with `InputStream`**: When `getPath()` or `getFile()` is called, the stream
- is dynamically buffered to memory first, then spills to a temporary file after a threshold.
- The temporary file is automatically cleaned up when the stream is closed.
+* **Initialized with `InputStream`**: `getPath()` or `getFile()` spools the stream to a
+ temporary file, which is automatically cleaned up when the stream is closed.
==== Benefits of Decentralized Spooling
@@ -73,14 +73,43 @@ File file = TikaInputStream.get(inputStream).getFile();
3. **Correctness**: Each component knows its own requirements.
4. **Flexibility**: New formats can be added without modifying central spooling logic.
-=== TikaInputStream Backing Strategies
+=== Rewinding and random access
-`TikaInputStream` uses configurable backing strategies that handle caching and temporary
-file management. This means:
+`Parser.parse` and `Detector.detect` take a `TikaInputStream`, so a caller with a
+plain `InputStream` must wrap it. What that wrapper can do depends on what backs it.
-* Repeated calls to `getFile()` return the same temporary file (no re-spooling).
-* The `rewind()` method efficiently resets the stream for re-reading.
-* Memory-mapped and disk-backed strategies can be selected based on use case.
+`Path`- and `byte[]`-backed streams are inherently rewindable: `rewind()`,
+`mark()`/`reset()`, `getPath()`, and `getFile()` always work, and
+`enableRewind()` is a no-op.
+
+An `InputStream`-backed stream starts in *passthrough* mode — a
+`BufferedInputStream`, so `mark()`/`reset()` work only within the buffer, and
+nothing is retained. Moving out of passthrough mode is only legal at position 0:
+
+[cols="2,2,3"]
+|===
+|Call |At position 0 |After reading
+
+|`enableRewind()`
+|Switches to caching mode: bytes are retained in memory, spilling to a temp file
+past 1 MB. `rewind()`, `mark()`/`reset()`, and random access all work afterwards.
+|`IllegalStateException`
+
+|`getPath()` / `getFile()`
+|Implicitly enables rewind, then spills to a temp file and returns its path.
+|`IOException` — "Cannot spill to file: position is N, must be 0"
+
+|`rewind()`
+|No-op.
+|`IOException` — "Cannot seek in passthrough mode. Call `enableRewind()` first."
+|===
+
+So: call `enableRewind()` **before reading a byte** whenever you will need to
+re-read the stream, take a digest, or hand it to something that needs a file.
+Skip it for streaming-only work (HTML, plain text) to avoid the caching cost.
+
+Repeated calls to `getPath()`/`getFile()` return the same temp file — spooling
+happens once — and the temp file is deleted when the stream is closed.
== User Guide
@@ -93,6 +122,10 @@ spool the input stream to a temporary file if necessary.
=== SpoolingStrategy for Fine-Grained Control
For advanced use cases, you can use `SpoolingStrategy` to control spooling behavior.
+
+NOTE: No detector or parser shipped with Tika consults `SpoolingStrategy` — it is an
+extension point for your own components.
+
This is useful when you want to:
* Restrict which file types are allowed to spool (e.g., for performance reasons)
@@ -117,7 +150,9 @@ ParseContext context = new ParseContext();
context.set(SpoolingStrategy.class, strategy);
// Parse with the custom context
-parser.parse(inputStream, handler, metadata, context);
+try (TikaInputStream tis = TikaInputStream.get(path)) {
+ parser.parse(tis, handler, metadata, context);
+}
----
==== SpoolingStrategy Methods
diff --git a/docs/modules/ROOT/pages/advanced/zip-detection.adoc b/docs/modules/ROOT/pages/advanced/zip-detection.adoc
index f362b0cc93..4d32a94f1e 100644
--- a/docs/modules/ROOT/pages/advanced/zip-detection.adoc
+++ b/docs/modules/ROOT/pages/advanced/zip-detection.adoc
@@ -55,7 +55,8 @@ You can check for this in your code:
----
import org.apache.tika.metadata.Zip;
-boolean wasSalvaged = metadata.getBoolean(Zip.SALVAGED);
+// getBoolean returns null when the property is not set
+boolean wasSalvaged = Boolean.TRUE.equals(metadata.getBoolean(Zip.SALVAGED));
----
=== Direct Parser Usage
@@ -72,26 +73,31 @@ For consistent behavior with truncated files, use `AutoDetectParser`:
[source,java]
----
-// Recommended: Uses detection with salvaging
+// Recommended: detection runs, so salvaging applies
AutoDetectParser parser = new AutoDetectParser();
-parser.parse(inputStream, handler, metadata, context);
+try (TikaInputStream tis = TikaInputStream.get(path)) {
+ parser.parse(tis, handler, metadata, context);
+}
----
Rather than calling parsers directly:
[source,java]
----
-// Direct parser: No salvaging support
+// Direct parser: no detection, so no salvaging -- may fail on truncated files
ZipParser parser = new ZipParser();
-parser.parse(inputStream, handler, metadata, context); // May fail on truncated files
+try (TikaInputStream tis = TikaInputStream.get(path)) {
+ parser.parse(tis, handler, metadata, context);
+}
----
== Other Detection Hints
The detector also sets additional metadata hints for parsers:
-`zip:detector-zip-file-opened`:: `true` if the detector successfully opened the ZIP as a `ZipFile`.
-The `ZipFile` object is available via `TikaInputStream.getOpenContainer()` for reuse by parsers.
+`zip:detector-zip-file-opened`:: `true` if the detector has a `ZipFile` for the parser to reuse —
+either the original or, when `zip:salvaged` is also `true`, the salvaged copy. The `ZipFile`
+object is available via `TikaInputStream.getOpenContainer()`.
`zip:detector-data-descriptor-required`:: `true` if streaming detection required
DATA_DESCRIPTOR support. This hint helps parsers choose the correct streaming mode.
diff --git a/docs/modules/ROOT/pages/configuration/detectors.adoc b/docs/modules/ROOT/pages/configuration/detectors.adoc
index 1ef2e1c058..9ff3e5b5e6 100644
--- a/docs/modules/ROOT/pages/configuration/detectors.adoc
+++ b/docs/modules/ROOT/pages/configuration/detectors.adoc
@@ -47,11 +47,11 @@ replaces the running type only with a strict specialization, so of two sibling r
the first — the opt-in detector must run first to win.
====
-With this configuration, `Tika.detect()` on a signed CMS message returns
+`Tika.detect()` on a signed CMS message then returns
`application/pkcs7-mime; smime-type=signed-data` instead of the coarse
-`application/pkcs7-signature`. This is verified end-to-end in
+`application/pkcs7-signature`, as verified end-to-end in
link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/ConfigExamplesTest.java[`ConfigExamplesTest`].
-NOTE: The default (magic-only) path already detects the crypto *families* — PKCS7/CMS,
-RFC 5544 timestamped-data, and PKCS12 — so most users do not need this detector. Enable it only
-when a routing or downstream decision depends on the CMS subtype before parsing.
+NOTE: The default magic-only path already identifies the crypto *families* — PKCS7/CMS, RFC 5544
+timestamped-data, PKCS12. Enable this detector only when a routing or downstream decision depends
+on the CMS subtype before parsing.
diff --git a/docs/modules/ROOT/pages/configuration/digesters.adoc b/docs/modules/ROOT/pages/configuration/digesters.adoc
index 67d8865451..931f824c98 100644
--- a/docs/modules/ROOT/pages/configuration/digesters.adoc
+++ b/docs/modules/ROOT/pages/configuration/digesters.adoc
@@ -17,27 +17,30 @@
= Configuring Digesters
-Tika can compute cryptographic digests (hashes) of documents during parsing. This is useful for
-document deduplication, integrity verification, and forensic analysis.
+Tika can hash documents during parsing — useful for deduplication, integrity verification and
+forensics. Digests land in metadata under `tk:digest:<ALGORITHM>` for the default HEX encoding, or
+`tk:digest:<ALGORITHM>:<ENCODING>` for the others, e.g. `tk:digest:SHA-256:BASE32`. The key
+carries the algorithm's JCA name (`SHA-256`, `SHA3-512`); config names it by the enum spelling
+below (`SHA256`, `SHA3_512`).
-== Overview
+Two implementations are available, both configured in the `parse-context` section:
-Digesters compute hash values of document content and store them in metadata. The digest value
-is stored with a key like `tk:digest:SHA256` (for HEX encoding) or `tk:digest:SHA256:BASE32`
-(for non-default encodings).
-
-Tika provides two digester implementations:
+[cols="2,1,3"]
+|===
+|Factory |Config key |Algorithms
-* **CommonsDigesterFactory** - Uses Apache Commons Codec. Supports MD2, MD5, SHA1, SHA256, SHA384, SHA512.
-* **BouncyCastleDigesterFactory** - Uses BouncyCastle provider. Supports all Commons algorithms plus SHA3-256, SHA3-384, SHA3-512.
+|`CommonsDigesterFactory`
+|`commons-digester-factory`
+|MD2, MD5, SHA1, SHA256, SHA384, SHA512 (Apache Commons Codec)
-== JSON Configuration
-
-Configure digesters in the `parse-context` section of your tika-config.json.
+|`BouncyCastleDigesterFactory`
+|`bouncy-castle-digester-factory`
+|All of the above plus SHA3_256, SHA3_384, SHA3_512 (BouncyCastle provider)
+|===
-=== Basic Example with CommonsDigester
+Encodings: `HEX` (default, lowercase), `BASE32` and `BASE64`, both RFC 4648.
-This example configures multiple digest algorithms:
+== JSON Configuration
.link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/configs/tika-config-commons-digests-basic.json[tika-config-commons-digests-basic.json]
[source,json]
@@ -55,10 +58,18 @@ This example configures multiple digest algorithms:
}
----
+Each entry takes an optional `encoding` — see
+link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/configs/tika-config-digests.json[tika-config-digests.json]:
+
+[source,json]
+----
+{ "algorithm": "SHA256", "encoding": "BASE32" }
+----
+
[#bouncycastle-sha3]
=== Using BouncyCastle for SHA3 Algorithms
-For SHA3 algorithms, use the BouncyCastle digester:
+Swap the factory key; SHA3 algorithms are spelled `SHA3_256`, `SHA3_384`, `SHA3_512`.
.link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/configs/tika-config-bc-digests-multiple.json[tika-config-bc-digests-multiple.json]
[source,json]
@@ -76,32 +87,10 @@ For SHA3 algorithms, use the BouncyCastle digester:
}
----
-=== Custom Encoding
-
-By default, digest values are encoded as lowercase hexadecimal. You can specify BASE32 or BASE64 encoding:
-
-.link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/configs/tika-config-digests.json[tika-config-digests.json]
-[source,json]
-----
-{
- "parse-context": {
- "commons-digester-factory": {
- "digests": [
- { "algorithm": "SHA256", "encoding": "BASE32" },
- { "algorithm": "MD5" }
- ]
- }
- }
-}
-----
-
-Non-default encodings include the encoding in the metadata key: `tk:digest:SHA256:BASE32`.
+=== Skipping the container digest
-=== Skip Container Document Digest
-
-When processing documents with embedded content (e.g., a ZIP file with PDFs inside), you may
-want to digest only the embedded documents, not the container. Set `skipContainerDocumentDigest`
-to `true`:
+To digest only the embedded documents of a container (say, the PDFs inside a zip) and not the
+container itself, set `skipContainerDocumentDigest` alongside `digests`:
.link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/configs/tika-config-digests-skip-container.json[tika-config-digests-skip-container.json]
[source,json]
@@ -109,77 +98,50 @@ to `true`:
{
"parse-context": {
"commons-digester-factory": {
- "digests": [
- { "algorithm": "MD5" }
- ],
+ "digests": [ { "algorithm": "MD5" } ],
"skipContainerDocumentDigest": true
}
}
}
----
-== Supported Algorithms
-
-[cols="1,1,1"]
-|===
-|Algorithm |CommonsDigester |BouncyCastleDigester
-
-|MD2 |Yes |Yes
-|MD5 |Yes |Yes
-|SHA1 |Yes |Yes
-|SHA256 |Yes |Yes
-|SHA384 |Yes |Yes
-|SHA512 |Yes |Yes
-|SHA3_256 |No |Yes
-|SHA3_384 |No |Yes
-|SHA3_512 |No |Yes
-|===
-
-== Supported Encodings
-
-* **HEX** (default) - Lowercase hexadecimal
-* **BASE32** - RFC 4648 Base32
-* **BASE64** - RFC 4648 Base64
-
== Programmatic Configuration
-You can also configure digesters programmatically via `ParseContext`:
-
[source,java]
----
-// See: CommonsDigesterFactory.java
CommonsDigesterFactory factory = new CommonsDigesterFactory();
factory.setDigests(Arrays.asList(
new DigestDef(DigestDef.Algorithm.SHA256),
- new DigestDef(DigestDef.Algorithm.MD5, DigestDef.Encoding.BASE32)
-));
+ new DigestDef(DigestDef.Algorithm.MD5, DigestDef.Encoding.BASE32)));
factory.setSkipContainerDocumentDigest(true);
ParseContext context = new ParseContext();
context.set(DigesterFactory.class, factory);
-// Use with AutoDetectParser
AutoDetectParser parser = new AutoDetectParser();
-parser.parse(inputStream, handler, metadata, context);
+try (TikaInputStream tis = TikaInputStream.get(inputStream)) {
+ parser.parse(tis, handler, metadata, context);
+}
----
-See link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-digest-commons/src/main/java/org/apache/tika/parser/digestutils/CommonsDigesterFactory.java[CommonsDigesterFactory.java] and
-link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-digest-commons/src/main/java/org/apache/tika/parser/digestutils/BouncyCastleDigesterFactory.java[BouncyCastleDigesterFactory.java] for implementation details.
-
-== Command Line Usage
+== Command Line
-When using the Tika CLI (`tika-app`), you can enable digesting with the `--digest` flag:
+`tika-app`'s `--digest=<algorithm>` flag covers the Commons algorithms only; the digest appears in
+the metadata output.
[source,bash]
----
java -jar tika-app.jar --digest=SHA256 document.pdf
----
-This computes a SHA256 digest of the document. The digest value appears in the metadata output.
+For SHA3, use a JSON config with `bouncy-castle-digester-factory` as above. See
+xref:using-tika/cli/index.adoc[the CLI reference].
-== Related Classes
+== Source
-* link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/DigesterFactory.java[DigesterFactory] - Factory interface
-* link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/DigestDef.java[DigestDef] - Algorithm and encoding definition
-* link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/Digester.java[Digester] - Digester interface
-* link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/digest/DigestConfigTest.java[DigestConfigTest] - Test examples
+* link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/DigesterFactory.java[`DigesterFactory`],
+ link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/Digester.java[`Digester`],
+ link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/digest/DigestDef.java[`DigestDef`]
+* link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-digest-commons/src/main/java/org/apache/tika/parser/digestutils/CommonsDigesterFactory.java[`CommonsDigesterFactory`],
+ link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-digest-commons/src/main/java/org/apache/tika/parser/digestutils/BouncyCastleDigesterFactory.java[`BouncyCastleDigesterFactory`]
+* link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/digest/DigestConfigTest.java[`DigestConfigTest`] — worked examples
diff --git a/docs/modules/ROOT/pages/configuration/encoding-detectors.adoc b/docs/modules/ROOT/pages/configuration/encoding-detectors.adoc
index 6a3c407cf5..01ed0f3355 100644
--- a/docs/modules/ROOT/pages/configuration/encoding-detectors.adoc
+++ b/docs/modules/ROOT/pages/configuration/encoding-detectors.adoc
@@ -17,119 +17,92 @@
= Configuring Encoding Detectors
-Tika uses a chain of _encoding detectors_ to determine the character encoding
-of plain text and HTML content. `DefaultEncodingDetector` discovers detectors
-via the Java service-provider interface (SPI, `META-INF/services`).
+Tika determines the character encoding of plain text and HTML with a chain of _encoding
+detectors_. `DefaultEncodingDetector` discovers them through the Java service-provider interface
+(`META-INF/services`).
-The chain runs in one of two modes:
+The chain runs in one of two modes, and which one you get depends on whether a
+`MetaEncodingDetector` is in it:
-* *collect-all* — when a `MetaEncodingDetector` is present (the 4.x default
- includes one), every base detector runs and contributes candidate encodings,
- then the meta detector picks the best one by decode quality. Registration
- order does not matter.
-* *first-match-wins* — otherwise, detectors run in registration order and the
- first non-null result is used.
+* *collect-all* — with a `MetaEncodingDetector` present (the default chain has one), every base
+ detector runs and contributes candidate encodings, and the meta detector picks the best by decode
+ quality. Registration order does not matter.
+* *first-match-wins* — otherwise, detectors run in registration order and the first non-null result
+ wins.
-== Default Detection Chain
+Because the default chain's `junk-filter-encoding-detector` is a `MetaEncodingDetector`, the stock
+distribution runs collect-all: order is irrelevant, and a declaration — a BOM, a `<meta charset>`
+tag — does *not* automatically win. The junk filter overrides either when the byte evidence
+strongly contradicts it.
-The stock 4.x distribution registers five detectors:
+== Available Detectors
+
+All implement `org.apache.tika.detect.EncodingDetector` and are referenced in JSON config by the
+name below.
-[cols="1,2,3"]
+[cols="2,2,1,3"]
|===
-|Detector |Module |Role
+|Name |Module |Default chain |Role
|`bom-detector`
|`tika-core`
+|Yes
|Emits a candidate from a leading byte-order mark.
|`metadata-charset-detector`
|`tika-core`
-|Emits a candidate from declarative hints (`Content-Type` charset,
-`Content-Encoding`) in the `Metadata` object.
-
-|`html-encoding-detector`
-|`tika-encoding-detector-html`
-|Emits a candidate from an HTML `<meta charset>` / `http-equiv` tag (lenient
-regex over a curated subset of WHATWG label aliases).
-
-|`mojibuster-encoding-detector`
-|`tika-encoding-detector-mojibuster`
-|Byte-bigram Naive Bayes classifier plus structural detectors for UTF-32 and
-UTF-16 and a UTF-8 grammar gate.
-
-|`junk-filter-encoding-detector`
-|`tika-ml-junkdetect`
-|`MetaEncodingDetector` that picks among the other detectors' candidates by
-script-aware decode quality. Always runs last.
-|===
-
-Because `junk-filter-encoding-detector` is a `MetaEncodingDetector`, the chain
-runs collect-all: detector order is irrelevant, and a declaration (a BOM or a
-`<meta charset>` tag) does *not* automatically win. The junk filter will
-override a declaration — or even a BOM — when the byte evidence strongly
-contradicts it.
-
-NOTE: This is a behaviour change from 3.x, whose default chain was
-`html` / `universal` / `icu4j` with first-match-wins (a declaration always
-won). `universal-encoding-detector` and `icu4j-encoding-detector` are no longer
-in the default distribution; see <<restore-3x,Restore the 3.x chain>>.
-
-== Available Detectors
-
-All detectors implement `org.apache.tika.detect.EncodingDetector` and can be
-referenced by their SPI name in JSON configuration.
-
-[cols="2,2,3"]
-|===
-|Name |Module |Description
-
-|`bom-detector`
-|`tika-core`
-|Reads a leading byte-order mark. In the default chain.
+|Yes
+|Emits a candidate from declarative hints (`Content-Type` charset, `Content-Encoding`) in the
+`Metadata` object.
-|`metadata-charset-detector`
+|`override-encoding-detector`
|`tika-core`
-|Reads declarative hints (`Content-Type` charset, `Content-Encoding`) from the
-`Metadata` object. In the default chain.
+|No
+|Always returns the configured charset, ignoring the document. Takes `charset` (default `UTF-8`).
+For a corpus whose encoding you already know.
|`html-encoding-detector`
|`tika-encoding-detector-html`
-|Fast lenient regex matcher for `<meta charset>` / `http-equiv` tags. In the
-default chain.
+|Yes
+|Emits a candidate from an HTML `<meta charset>` / `http-equiv` tag, via a lenient regex over a
+curated subset of WHATWG label aliases.
|`mojibuster-encoding-detector`
|`tika-encoding-detector-mojibuster`
-|Byte-bigram Naive Bayes classifier with structural UTF-32/UTF-16 detectors and
-a UTF-8 grammar gate. In the default chain.
+|Yes
+|Byte-bigram Naive Bayes classifier, plus structural detectors for UTF-32 and UTF-16 and a UTF-8
+grammar gate.
|`junk-filter-encoding-detector`
|`tika-ml-junkdetect`
-|Text-quality arbitrator (`MetaEncodingDetector`). In the default chain; runs
-last.
+|Yes, last
+|`MetaEncodingDetector` that picks among the other detectors' candidates by script-aware decode
+quality.
|`standard-html-encoding-detector`
|`tika-encoding-detector-html`
-|Spec-strict WHATWG prescan algorithm. Not in the default chain — opt in if you
-need strict WHATWG tokenisation (e.g. ignoring charset declarations inside HTML
-comments).
+|No
+|Spec-strict WHATWG prescan. Opt in when you need strict WHATWG tokenisation — for example,
+ignoring charset declarations inside HTML comments.
|`universal-encoding-detector`
|`tika-encoding-detector-universal`
-|State-machine structural prober (juniversalchardet fork). Not bundled and not
-auto-discovered; add the jar and configure it explicitly to use it.
+|No
+|State-machine structural prober (juniversalchardet fork). Not bundled and not auto-discovered: add
+the jar *and* configure it explicitly.
|`icu4j-encoding-detector`
|`tika-encoding-detector-icu4j`
-|Wraps ICU4J's `CharsetDetector`. Not bundled and not auto-discovered; add the
-jar and configure it explicitly to use it.
+|No
+|Wraps ICU4J's `CharsetDetector`. Not bundled and not auto-discovered: add the jar *and* configure
+it explicitly.
|===
== Configuration Examples
=== Exclude a detector from the default chain
-Use `default-encoding-detector` with an `exclude` list to drop one or more
-auto-registered detectors:
+`default-encoding-detector` takes an `exclude` list of auto-registered detectors to drop:
[source,json]
----
@@ -144,18 +117,15 @@ auto-registered detectors:
}
----
-NOTE: Do not combine `default-encoding-detector` with other explicit detector
-entries in the same list. When combined, the loader wraps everything in an
-outer composite that has no `MetaEncodingDetector` at its top level, so
-collect-all arbitration is silently lost and the explicit detectors are never
-reached. Use an explicit chain (see below) when you need to configure
-individual detectors.
+WARNING: Do not combine `default-encoding-detector` with explicit detector entries in the same
+list. The loader then wraps everything in an outer composite with no `MetaEncodingDetector` at its
+top level, so collect-all arbitration is silently lost and the explicit detectors are never
+reached. To configure individual detectors, specify the whole chain explicitly instead.
=== Specify the chain explicitly
-To replace the SPI-discovered chain with an explicit ordered list. Include
-`junk-filter-encoding-detector` (last) to keep collect-all arbitration; omit it
-for first-match-wins:
+An explicit ordered list replaces the SPI-discovered chain. End it with
+`junk-filter-encoding-detector` to keep collect-all arbitration; omit it for first-match-wins.
[source,json]
----
@@ -170,16 +140,13 @@ for first-match-wins:
=== Configure the HTML detector's read limit
-`html-encoding-detector` reads up to 65 536 bytes by default when scanning for
-the `<meta charset>` tag. Raise it if your documents embed large `<script>`
-blocks before the meta tag (TIKA-2485). (`mojibuster-encoding-detector` reads a
-larger content probe, so in the default chain this limit matters mainly for very
-large preambles.)
+`html-encoding-detector` scans the first `markLimit` bytes for the `<meta charset>` tag, 8192 by
+default. Raise it if your documents put large `<script>` blocks before the meta tag (TIKA-2485).
+`mojibuster-encoding-detector` reads a larger content probe, so in the default chain this limit
+matters mainly for very large preambles.
-To configure `markLimit`, specify the full chain explicitly. An explicit list
-that includes `junk-filter-encoding-detector` keeps collect-all arbitration; the
-configured `html-encoding-detector` participates as a base detector alongside
-Mojibuster, and the junk filter arbitrates as usual:
+Configuring it means writing the chain out explicitly; the configured detector then participates as
+a base detector alongside Mojibuster, and the junk filter arbitrates as usual:
[source,json]
----
@@ -194,8 +161,8 @@ Mojibuster, and the junk filter arbitrates as usual:
=== Use the spec-strict WHATWG HTML detector
-If your input HTML has charset declarations inside comments (or other contexts
-where the lenient regex would false-match), opt in to the spec-strict prescan:
+Swap in `standard-html-encoding-detector` when the lenient regex would false-match — charset
+declarations inside comments, for instance:
[source,json]
----
@@ -211,17 +178,17 @@ where the lenient regex would false-match), opt in to the spec-strict prescan:
[#restore-3x]
=== Restore the 3.x detection chain (universal + icu4j)
-The 4.x default no longer bundles or auto-registers
-`universal-encoding-detector` and `icu4j-encoding-detector`. To get the legacy
-3.x behaviour (`html` / `universal` / `icu4j`, first-match-wins) you must do
-*both*:
+`universal-encoding-detector` and `icu4j-encoding-detector` are neither bundled nor
+auto-registered. To get the 3.x chain (`html` / `universal` / `icu4j`, first-match-wins) you must
+do *both*:
-. *Add the jars to the classpath.* They are no longer in the `tika-app` /
-`tika-server-standard` packages, so supply `tika-encoding-detector-universal`
-and `tika-encoding-detector-icu4j` yourself (for example via `-Dtika.extras.dir`
-— see xref:configuration/index.adoc[the configuration overview]).
-. *Configure the chain explicitly.* An explicit chain with no
-`MetaEncodingDetector` runs first-match-wins:
+. *Add the jars to the classpath.* They are not in the `tika-app` or `tika-server-standard`
+packages, so supply `tika-encoding-detector-universal` and `tika-encoding-detector-icu4j` yourself
+— for example via `-Dtika.extras.dir`, see xref:configuration/index.adoc[the configuration
+overview].
+. *Configure the chain explicitly.* Dropping the jars on the classpath is not enough: unlike the
+other detectors, these two are config-only. An explicit chain with no `MetaEncodingDetector` runs
+first-match-wins.
[source,json]
----
@@ -233,6 +200,3 @@ and `tika-encoding-detector-icu4j` yourself (for example via `-Dtika.extras.dir`
]
}
----
-
-Dropping the jars on the classpath alone is *not* enough: unlike the other
-detectors, these two are config-only and are not auto-discovered via SPI.
diff --git a/docs/modules/ROOT/pages/configuration/index.adoc b/docs/modules/ROOT/pages/configuration/index.adoc
index afd6f35fd7..0658120050 100644
--- a/docs/modules/ROOT/pages/configuration/index.adoc
+++ b/docs/modules/ROOT/pages/configuration/index.adoc
@@ -18,49 +18,17 @@
= Configuration
:toc:
-This section covers configuring Apache Tika.
+Tika 4.x is configured with a JSON file — parsers, detectors, content handlers, server behavior and
+the Tika Pipes pipeline. `//` and `/* */` comments are allowed.
-== Overview
-
-Tika 4.x uses JSON configuration files. Configuration controls parsers, detectors,
-content handlers, server behavior, and the Tika Pipes pipeline.
-
-NOTE: Tika 3.x and earlier used XML configuration (`tika-config.xml`). See the
-xref:migration-to-4x/index.adoc[Migration Guide] for details on converting to JSON.
-
-== Adding extra jars (`tika.extras.dir`)
-
-To add extra components — additional `EncodingDetector` or `Parser`
-implementations, or their dependencies — without repackaging the application,
-drop their jars in a directory and point the `tika.extras.dir` system property
-at it:
-
-[source,bash]
-----
-java -Dtika.extras.dir=/path/to/extras -jar tika-app.jar ...
-java -Dtika.extras.dir=/path/to/extras -jar tika-server-standard.jar ...
-----
-
-Every `*.jar` in that directory is added to the classpath that Tika's
-service-loading scans, so SPI-registered components in those jars are picked up
-automatically. The jars are also forwarded onto forked Pipes/server worker
-processes, so they are available where parsing happens.
-
-This is *off by default* — nothing is loaded unless `tika.extras.dir` is set, and
-there is no implicit default directory (Tika does not scan the working directory automatically). Treat the directory as a trusted code
-location: anything in it runs with the full privileges of the Tika process, so it
-must not be writable by less-trusted principals (and, for a server, must not be
-reachable by request handling).
-
-NOTE: In the Docker images you can instead mount a directory to `/tika-extras`.
-PF4J-managed plugins are a separate mechanism — see
-xref:pipes/plugins/index.adoc[Pipes Plugins].
+NOTE: Converting a 3.x `tika-config.xml`? See the
+xref:migration-to-4x/index.adoc[Migration Guide], and `tika-app`'s
+xref:using-tika/cli/index.adoc[`--convert-config-xml-to-json`] flag.
== Top-level JSON structure
-A `tika-config.json` is a single JSON object whose keys are the top-level sections
-listed below. Every section is optional — omit what you don't need. Defaults are
-used wherever a section is missing.
+A `tika-config.json` is a single JSON object whose keys are the sections below. Every section is
+optional; anything you omit uses its defaults.
[source,json]
----
@@ -68,37 +36,73 @@ used wherever a section is missing.
"parsers": [ /* parser declarations */ ],
"detectors": [ /* detector declarations */ ],
"encoding-detectors": [ /* encoding detector declarations */ ],
+ "metadata-filters": [ /* metadata filter declarations */ ],
+ "renderers": [ /* page renderer declarations */ ],
+ "translator": { /* translator declaration */ },
"content-handler-factory": { /* handler type for emitted content */ },
+ "auto-detect-parser": { /* AutoDetectParser options */ },
"parse-context": {
"timeout-limits": { /* progress + total task timeouts */ },
"unpack-config": { /* embedded-byte extraction */ }
/* other SelfConfiguring components by component name */
},
- "server": { /* tika-server options: allowPipes, allowPerRequestConfig, cors, ... */ },
+ "server": { /* tika-server options: allowPipes, allowPerRequestConfig, cors, tlsConfig, ... */ },
+ "grpc": { /* tika-grpc options */ },
"pipes": { /* Pipes process management: numClients, parseMode, ... */ },
"fetchers": { /* named fetcher instances */ },
"emitters": { /* named emitter instances */ },
"pipes-iterator": { /* iterator (one per pipeline) */ },
"pipes-reporters": { /* per-document status reporters */ },
- "plugin-roots": "/path/to/plugins"
+ "plugin-roots": "/path/to/plugins",
+ "metadata-list": { /* Jackson read limits for metadata JSON */ },
+ "service-loader": { /* SPI load-failure policy */ },
+ "xml-reader-utils": { /* XML parser pool size */ }
}
----
-Per-section documentation:
-
-* `parsers`, `detectors`, `encoding-detectors`, `content-handler-factory`,
- `parse-context` — covered below under <<_topics,Topics>>.
-* `server` — see xref:using-tika/server/index.adoc[Tika Server].
-* `pipes`, `fetchers`, `emitters`, `pipes-iterator`, `pipes-reporters`,
- `plugin-roots` — see xref:pipes/configuration.adoc[Pipes Configuration]
- and xref:pipes/index.adoc[Tika Pipes].
+That list is exhaustive: an unrecognized top-level key is fatal at load time, and the error names
+the valid ones.
+
+`parsers`, `detectors`, `encoding-detectors`, `metadata-filters`, `content-handler-factory` and
+`parse-context` are covered under <<_topics,Topics>> below. For `server` see
+xref:using-tika/server/index.adoc[Tika Server]; for `grpc` see
+xref:using-tika/grpc/index.adoc[Tika gRPC]; for `pipes`, `fetchers`, `emitters`,
+`pipes-iterator`, `pipes-reporters` and `plugin-roots` see
+xref:pipes/configuration.adoc[Pipes Configuration].
+
+[#typos]
+=== Where a typo is caught, and where it is not
+
+Unknown keys are rejected almost everywhere: at the top level, inside a component's config body,
+and inside a `FetchEmitTuple`. Four places swallow them silently instead, so a misspelling there
+produces no error and no effect:
+
+* the `metadata-list` body — only `maxStringLength`, `maxNestingDepth` and `maxNumberLength` are
+ read;
+* the body of a self-configuring component under `parse-context` — it is handed to the component
+ untouched;
+* `_mime-include` / `_mime-exclude` when the value is not a JSON array — a shape error yields an
+ empty filter rather than a failure;
+* an unregistered component name inside `default-parser`'s `exclude` list — logged as a `WARN`
+ only. (An unknown *key* in that same block is fatal.)
+
+If a setting appears to do nothing, check its spelling against one of those four first.
+
+[#component-names]
+== Component names
+
+Everything you configure is referenced by its *component name* — a kebab-case identifier such as
+`pdf-parser`, `file-system-fetcher` or `unpack-config`, derived from the class's simple name unless
+the component declares its own. (`developers/` calls the derived form a *friendly name*; it is the
+same string.) Class names are not accepted in their place: only components registered with
+`@TikaComponent` can be instantiated from JSON, which is what stops a config file from loading
+arbitrary classes. `tika-app --list-parser-names` prints the registered parser names, and an
+unknown name fails at load time listing the ones that are registered.
== The `parsers` list and `default-parser`
-Tika configuration files are JSON, with optional support for `//` and `/* */` comments.
-
-A `parsers` list loads *only* the parsers it names — every other parser is dropped.
-To customize one parser while keeping all the others, add a `default-parser` entry:
+A `parsers` list loads *only* the parsers it names — every other parser is dropped. To customize
+one parser while keeping all the others, add a `default-parser` entry:
[source,json]
----
@@ -110,32 +114,62 @@ To customize one parser while keeping all the others, add a `default-parser` ent
}
----
-Configuring a parser automatically excludes its default copy, so there is no duplication.
-Omit `default-parser` only when you want a Tika limited to the parsers you listed.
+Configuring a parser automatically excludes its default copy, so there is no duplication. Omit
+`default-parser` only when you want a Tika limited to the parsers you listed.
+
+`detectors` works the same way, with a `default-detector` entry. `encoding-detectors` has a
+`default-encoding-detector`, but it must not be mixed with explicit detector entries — see
+xref:configuration/encoding-detectors.adoc[Encoding Detectors].
== Windows file paths
-JSON uses the backslash as an escape character, so path options (e.g. `tesseractPath`,
-`imageMagickPath`) must use forward slashes (`C:/Tools/...`) or escaped backslashes
+JSON treats the backslash as an escape character, so path options (`tesseractPath`,
+`imageMagickPath`, ...) need forward slashes (`C:/Tools/...`) or escaped backslashes
(`C:\\Tools\\...`). A single backslash is a JSON parse error.
+== Adding extra jars (`tika.extras.dir`)
+
+To add components — extra `EncodingDetector` or `Parser` implementations, or their dependencies —
+without repackaging the application, drop their jars in a directory and point the
+`tika.extras.dir` system property at it:
+
+[source,bash]
+----
+java -Dtika.extras.dir=/path/to/extras -jar tika-app.jar ...
+java -Dtika.extras.dir=/path/to/extras -jar tika-server-standard.jar ...
+----
+
+Every `*.jar` in that directory joins the classpath Tika's service loading scans, so SPI-registered
+components are picked up automatically. The jars are forwarded to the forked JVMs that Pipes and
+tika-server run, too, so they are present where parsing happens.
+
+This is off by default: nothing is loaded unless `tika.extras.dir` is set, and there is no implicit
+default directory. Treat the directory as a trusted code location — anything in it runs with the
+full privileges of the Tika process, so it must not be writable by less-trusted principals, nor (on
+a server) reachable by request handling.
+
+NOTE: In the Docker images, mount a directory to `/tika-extras` instead. PF4J-managed plugins are a
+separate mechanism — see xref:pipes/plugins/index.adoc[Pipes Plugins].
+
+[#_topics]
== Topics
-=== Parser Configuration
+Parser configuration:
* xref:configuration/parsers/pdf-parser.adoc[PDFParser] — PDF parsing options
-* xref:configuration/parsers/tesseract-ocr-parser.adoc[TesseractOCRParser] — OCR options for image-based text extraction
-* xref:configuration/parsers/tess4j-parser.adoc[Tess4J OCR Parser] — in-process OCR via tess4j JNA bindings (advanced users only; most users should prefer the TesseractOCRParser above)
+* xref:configuration/parsers/tesseract-ocr-parser.adoc[TesseractOCRParser] — OCR for image-based
+ text extraction
+* xref:configuration/parsers/tess4j-parser.adoc[Tess4J OCR Parser] — in-process OCR via tess4j JNA
+ bindings; advanced users only, most should prefer `TesseractOCRParser`
* xref:configuration/parsers/vlm-parsers.adoc[VLM Parsers] — Claude, Gemini, OpenAI, Ollama, vLLM
-* xref:configuration/parsers/external-parser.adoc[External Parser] — wrap external tools (ffmpeg, exiftool, etc.)
-
-=== Other Configuration
+* xref:configuration/parsers/external-parser.adoc[External Parser] — wrap external tools (ffmpeg,
+ exiftool, ...)
-* xref:configuration/detectors.adoc[Detectors] — Configuring content (MIME) detection, incl. the opt-in PKCS7/CMS detector
-* xref:configuration/digesters.adoc[Digesters] — Computing cryptographic hashes of documents
-* xref:configuration/encoding-detectors.adoc[Encoding Detectors] — Configuring charset/encoding detection
+Other configuration:
-// Add links to specific topics as they are created
-// * xref:json-config.adoc[JSON Configuration Reference]
-// * xref:detectors.adoc[Configuring Detectors]
-// * xref:mime-types.adoc[MIME Type Configuration]
+* xref:configuration/metadata-filters.adoc[Metadata Filters] — trim, rename and restrict the
+ metadata that leaves the parse
+* xref:configuration/detectors.adoc[Detectors] — content (MIME) detection, including the opt-in
+ PKCS7/CMS detector
+* xref:configuration/digesters.adoc[Digesters] — cryptographic hashes of documents
+* xref:configuration/encoding-detectors.adoc[Encoding Detectors] — charset detection
diff --git a/docs/modules/ROOT/pages/configuration/metadata-filters.adoc b/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
new file mode 100644
index 0000000000..c6eee220fa
--- /dev/null
+++ b/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
@@ -0,0 +1,254 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+= Metadata Filters
+:toc:
+
+A metadata filter rewrites the metadata *after* parsing and *before* the result is emitted or
+serialized — the last chance to drop a field, rename it, normalize it, or add a derived one.
+Filters are the supported way to shape Tika's output; parsers are not the place to do it.
+
+Filters run in `tika-app`, `tika-server` and Tika Pipes alike, on the whole metadata list
+(container document first, then each embedded document).
+
+== The `metadata-filters` section
+
+`metadata-filters` is a top-level array. The filters run in the order listed, each one seeing what
+the previous one produced.
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ { "include-field-metadata-filter": { "include": ["tk:content", "Content-Type"] } },
+ "date-normalizing-metadata-filter"
+ ]
+}
+----
+
+An element is either a *string* (the filter's name, default settings) or an *object* keyed by that
+name with its configuration.
+
+IMPORTANT: There is no `default-metadata-filter` and no implicit filtering. If you omit the section
+— or give it an empty array — *no* filter runs and the metadata is emitted unchanged. This is
+unlike `parsers` and `detectors`, where a `default-parser` / `default-detector` entry restores the
+components you did not name.
+
+== Built-in filters
+
+These ship in `tika-core` and need no extra dependency.
+
+[cols="2,3,2"]
+|===
+|Name |What it does |Options (default)
+
+|`include-field-metadata-filter`
+|Keeps only the named fields; drops everything else.
+|`include` — list of field names (empty)
+
+|`exclude-field-metadata-filter`
+|Drops the named fields; keeps everything else.
+|`exclude` — list of field names (empty)
+
+|`field-name-mapping-filter`
+|Renames fields. With `excludeUnmapped`, doubles as an allow-list that renames as it filters.
+|`mappings` — map of old name to new name (empty); `excludeUnmapped` — drop fields with no
+mapping (`true`)
+
+|`remove-by-mime-metadata-filter`
+|Removes a document's entire `Metadata` object from the list if its `Content-Type` base type
+matches. Use it to keep, say, embedded images out of the output entirely.
+|`mimes` — list of base media types, no parameters (empty)
+
+|`clear-by-attachment-type-metadata-filter`
+|Empties (but keeps) the `Metadata` of an embedded document whose
+`tk:embedded-resource-type` matches.
+|`types` — list of `INLINE`, `ATTACHMENT`, `MACRO`, `METADATA`, `FONT`, `THUMBNAIL`,
+`RENDERING`, `VERSION`, `ALTERNATE_FORMAT_CHUNK` (empty). An unrecognized name is a config error.
+
+|`date-normalizing-metadata-filter`
+|Rewrites every `DATE`-typed field that has no timezone to `yyyy-MM-dd'T'HH:mm:ss'Z'`, assuming
+the configured zone. Values that already end in `Z` are left alone.
+|`defaultTimeZone` — zone id, e.g. `America/New_York` (`UTC`)
+
+|`capture-group-metadata-filter`
+|Runs a regex against the *first* value of `sourceField` and writes capture group 1 to
+`targetField`, overwriting it. No match, or a missing source field, is a no-op.
+|`regex`, `sourceField`, `targetField` — all required
+
+|`geo-point-metadata-filter`
+|If both latitude and longitude are present, writes `"<lat>,<lon>"` to one field, the shape
+Elasticsearch and Solr want for a geo point.
+|`geoPointFieldName` (`location`)
+
+|`legacy-key-migration-filter`
+|Rewrites 4.x `tk:` keys back to their 3.x spellings (or forward). See
+xref:migration-to-4x/metadata-changes-4x.adoc[Metadata Changes in 4.x].
+|`direction` (`V4_TO_V3`); `table` — classpath resource for the rename table
+
+|`no-op-filter`
+|Does nothing. Only useful as an explicit "filter nothing here".
+|none
+|===
+
+== Filters from optional modules
+
+Adding the module to the classpath is enough to make the name resolvable.
+
+[cols="2,2,3"]
+|===
+|Name |Module |What it adds
+
+|`openai-embedding-filter`, `jina-embedding-filter`
+|`tika-inference`
+|Chunks `tk:content` and calls an embeddings endpoint, putting the vectors on `tk:chunks`. See
+xref:migration-to-4x/chunk-strategies.adoc[Chunk Strategies].
+
+|`charsoup-metadata-filter`, `optimaize-metadata-filter`, `open-nlp-metadata-filter`
+|`tika-langdetect-charsoup`, `-optimaize`, `-opennlp`
+|Detects the language of the extracted text and writes it to the metadata. See
+xref:advanced/language-detection.adoc[Language Detection].
+
+|`tika-eval-metadata-filter`
+|`tika-eval-core`
+|Scores extraction quality per document: `tika-eval:numTokens`, `numUniqueTokens`,
+`numAlphaTokens`, `numCommonTokens`, `numUniqueAlphaTokens`, `lang`, `langConfidence`, `oov`,
+`languageness`. Useful for spotting garbage extractions before they reach an index.
+|===
+
+== Common tasks
+
+=== Emit only the fields your index needs
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ {
+ "include-field-metadata-filter": {
+ "include": ["tk:content", "Content-Type", "dc:title", "dc:creator"]
+ }
+ }
+ ]
+}
+----
+
+=== Rename fields for a downstream schema
+
+`excludeUnmapped` defaults to `true`, so this keeps *only* the four mapped fields, under their new
+names:
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ {
+ "field-name-mapping-filter": {
+ "excludeUnmapped": true,
+ "mappings": {
+ "tk:content": "content",
+ "Content-Type": "mime",
+ "dc:title": "title",
+ "dc:creator": "author"
+ }
+ }
+ }
+ ]
+}
+----
+
+=== Strip the parameters off a media type
+
+`Content-Type` often arrives as `text/plain; charset=UTF-8`. To index the base type only:
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ {
+ "capture-group-metadata-filter": {
+ "sourceField": "Content-Type",
+ "targetField": "Content-Type",
+ "regex": "\\A([^;]+)"
+ }
+ }
+ ]
+}
+----
+
+=== Drop the metadata of embedded images
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ { "clear-by-attachment-type-metadata-filter": { "types": ["INLINE"] } }
+ ]
+}
+----
+
+To remove those documents from the list altogether rather than blank them, filter by media type
+instead:
+
+[source,json]
+----
+{
+ "metadata-filters": [
+ { "remove-by-mime-metadata-filter": { "mimes": ["image/jpeg", "image/png"] } }
+ ]
+}
+----
+
+== Ordering
+
+Composition is literal: filter _n_ sees filter _n-1_'s output. Two consequences worth planning
+around.
+
+* An allow-list runs last, or it removes the fields a later filter wanted. `include-field-metadata-filter`
+ before `geo-point-metadata-filter` drops the latitude and longitude the geo filter reads — unless
+ you included them.
+* A rename runs last, or later filters have to be configured against the new names.
+
+== Where filters do and do not apply
+
+* `tika-app` applies them to the metadata list before writing output.
+* `tika-server`'s `/tika`, `/rmeta`, `/meta` and `/unpack` parse in forked JVMs, and the filters
+ configured in the server's `tika-config.json` are applied there before the response is written.
+* Tika Pipes applies them at the emit edge, before the emitter runs.
+
+`parse-context` can carry a per-request `metadata-filters` list, which *replaces* the configured
+one for that request rather than adding to it.
+
+NOTE: In `CONTENT_ONLY` xref:pipes/parse-modes.adoc[parse mode], Tika applies an
+`include-field-metadata-filter` for `tk:content` and `tk:exception:container-exception` when you have
+configured no filter of your own. Configure one and yours wins — including having to keep
+`tk:content` yourself.
+
+== Writing your own
+
+`MetadataFilter` is an extension point. Extend `MetadataFilterBase` if one `Metadata` at a time is
+enough, or `MetadataFilter` if you need the whole list; annotate with `@TikaComponent` to get a
+config name. Writing a reserved `tk:` key by name (rather than through its `Property`) requires
+`Metadata#setTrusted` / `addTrusted`. See
+xref:developers/serialization.adoc[Serialization and Configuration] for a worked example.
+
+== Source
+
+* link:https://github.com/apache/tika/tree/main/tika-core/src/main/java/org/apache/tika/metadata/filter[`org.apache.tika.metadata.filter`]
+ — the built-in filters
+* link:https://github.com/apache/tika/blob/main/tika-core/src/main/java/org/apache/tika/metadata/filter/CompositeMetadataFilter.java[`CompositeMetadataFilter`]
+ — the ordering semantics above
diff --git a/docs/modules/ROOT/pages/configuration/parsers/external-parser.adoc b/docs/modules/ROOT/pages/configuration/parsers/external-parser.adoc
index 5e4002545a..f9eed90ea2 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/external-parser.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/external-parser.adoc
@@ -17,9 +17,8 @@
= External Parser Configuration
-The `ExternalParser` allows Tika to delegate parsing to external command-line
-programs such as `ffmpeg`, `exiftool`, or `sox`. Each external parser is
-configured via JSON and must be explicitly enabled -- Tika 4.x does not
+`ExternalParser` delegates parsing to external command-line programs such as `ffmpeg`, `exiftool`
+or `sox`. Each one is configured in JSON and must be enabled explicitly — Tika does not
auto-discover external tools at startup.
== Key Concepts
@@ -61,6 +60,11 @@ configured, the raw bytes are written as text.
|===
|Field |Type |Description
+|`_mime-include`
+|`List<String>`
+|Media types this parser handles. (`_mime-exclude` is the inverse; both are generic parser
+decorators, not specific to `ExternalParser`.)
+
|`commandLine`
|`List<String>`
|The command and arguments to run. Use `$\{INPUT_FILE}` and `$\{OUTPUT_FILE}` tokens for file paths.
@@ -95,7 +99,7 @@ configured, the raw bytes are written as text.
|`returnStderr`
|`boolean`
-|Store raw stderr in metadata. Default: `true`.
+|Store raw stderr in metadata. Default: `false`.
|`timeoutMillis`
|`long`
@@ -146,9 +150,8 @@ icon:github[] https://github.com/apache/tika/blob/main/tika-serialization/src/te
=== Multiple External Parsers
-You can configure multiple external parsers in a single config file. Each
-handles different MIME types via `_mime-include`. Here FFmpeg handles video
-files while exiftool handles PDFs:
+A single config file can hold several, each scoped to different media types by `_mime-include`.
+Here FFmpeg handles video while exiftool handles PDFs:
[source,json]
----
@@ -156,21 +159,6 @@ include::example$external-parser-multi.json[]
----
icon:github[] https://github.com/apache/tika/blob/main/tika-serialization/src/test/resources/configs/external-parser-multi.json[View source on GitHub]
-== Changes from 3.x
-
-In Tika 3.x, external parsers were configured via XML (`tika-external-parsers.xml`)
-and auto-discovered at startup. The `CompositeExternalParser` would fork
-a process for each configured tool on every Tika initialization to check
-if the tool was available.
-
-In Tika 4.x:
-
-* External parsers must be explicitly configured in JSON -- no auto-discovery.
-* The `checkCommandLine` runs lazily on first use, not at startup.
-* Three independent stream handlers (`stdoutHandler`, `stderrHandler`,
- `outputFileHandler`) replace the old `outputParser`/`stderrParser` split.
-* The `contentSource` field explicitly controls which stream provides text content.
-* `CompositeExternalParser`, `ExternalParsersFactory`, and the XML config
- reader have been removed.
-
-See xref:migration-to-4x/migrating-to-4x.adoc[Migrating to 4.x] for general migration guidance.
+Converting a 3.x `tika-external-parsers.xml`? See
+xref:migration-to-4x/migrating-to-4x.adoc[Migrating to 4.x], which covers the removal of
+`CompositeExternalParser`, `ExternalParsersFactory` and XML-based auto-discovery.
diff --git a/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc b/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
index 63ffa53c7c..e15ce5f6f5 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
@@ -17,7 +17,7 @@
= PDFParser Configuration
-This page documents the configuration options for `PDFParser` in Tika 4.x.
+Configuration options for `PDFParser`.
== Basic Configuration
@@ -29,16 +29,12 @@ icon:github[] https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers
== Full Configuration
-The example below lists every option with its default value and an inline comment describing
-it. It also includes a `default-parser` entry so the config works as-is; see
-xref:configuration/index.adoc[Configuration] for why that entry matters.
+Every option with its default value and an inline comment describing it. The `default-parser` entry
+keeps the rest of Tika's parsers loaded — see xref:configuration/index.adoc[Configuration] for why
+that matters.
[source,json]
----
include::example$pdf-parser-full.json[]
----
icon:github[] https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/pdf-parser-full.json[View source on GitHub]
-
-== Changes from 3.x
-
-See xref:migration-to-4x/migrating-to-4x.adoc[Migrating to 4.x] for general migration guidance.
diff --git a/docs/modules/ROOT/pages/configuration/parsers/tess4j-parser.adoc b/docs/modules/ROOT/pages/configuration/parsers/tess4j-parser.adoc
index 8f82a0fd39..3f37619dba 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/tess4j-parser.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/tess4j-parser.adoc
@@ -19,57 +19,41 @@
[IMPORTANT]
====
-*Advanced users only.* `Tess4JParser` loads the Tesseract native library
-directly into your JVM through https://github.com/java-native-access/jna[JNA]
-(Java Native Access). Operating it safely means locating and linking the
-correct platform-specific native libraries, reasoning about the Java/native
-boundary, and accepting that a fault in the native code can crash the entire
-JVM. In short, it assumes you are comfortable working with native-library
-integration via JNA.
-
-If that doesn't describe you, please don't reach for this parser — and that's
-perfectly fine. The standard
-xref:configuration/parsers/tesseract-ocr-parser.adoc[`TesseractOCRParser`]
-performs the same OCR by running the `tesseract` command-line program in a
-separate process. It needs no native linking, is far easier to set up, and a
-crash in Tesseract can never take down your application, so it is the
-recommended choice for almost everyone. Choose `Tess4JParser` only when you
-have a measured need for in-process OCR throughput *and* the expertise to run
-native bindings safely.
+*Advanced users only.* `Tess4JParser` loads the Tesseract native library into your JVM through
+https://github.com/java-native-access/jna[JNA]. Running it safely means locating and linking the
+right platform-specific native libraries, reasoning about the Java/native boundary, and accepting
+that a fault in native code can crash the whole JVM.
+
+If that doesn't describe you, don't reach for this parser — and that's perfectly fine.
+xref:configuration/parsers/tesseract-ocr-parser.adoc[`TesseractOCRParser`] does the same OCR by
+running the `tesseract` command-line program in a separate process: no native linking, far easier
+to set up, and a Tesseract crash can never take down your application. Choose `Tess4JParser` only
+with a measured need for in-process OCR throughput *and* the expertise to run native bindings
+safely.
====
-The `Tess4JParser` is an OCR parser that calls the Tesseract native library
-in-process via https://github.com/nguyenq/tess4j[Tess4J] and JNA, rather
-than spawning a `tesseract` child process for every image. This eliminates
-per-file process-spawn overhead and can be significantly faster when
-processing large batches of images.
+`Tess4JParser` calls the Tesseract native library in-process via
+https://github.com/nguyenq/tess4j[Tess4J] and JNA instead of spawning a `tesseract` child process
+per image. That removes the per-file process-spawn overhead and can be significantly faster on
+large batches.
-Because the native Tesseract handle is *not thread-safe*, the parser
-maintains a configurable pool of `Tesseract` instances. Multiple threads
-borrow from the pool and return instances when done, so the parser is safe
-for concurrent use.
+The native Tesseract handle is *not thread-safe*, so the parser keeps a configurable pool of
+`Tesseract` instances that threads borrow and return; the parser itself is safe for concurrent use.
-[WARNING]
-====
-This parser loads native C/{cpp} libraries (Tesseract, Leptonica) into
-the JVM via JNA. A segfault or memory leak in the native code *will*
-crash your entire JVM.
-
-*You should run this parser in a forked child process using tika-pipes,
-ideally inside a Docker container.* Do not load it into a long-lived
-application server process unless you are comfortable with the risk.
-====
+WARNING: *Run this parser in a forked JVM using tika-pipes, ideally inside a Docker
+container.* A segfault or memory leak in the native Tesseract or Leptonica code takes down whatever
+JVM loaded it. Do not put it in a long-lived application server process.
== Module dependency
The parser lives in the `tika-parser-tess4j-module` artifact:
-[source,xml]
+[source,xml,subs=attributes+]
----
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-parser-tess4j-module</artifactId>
- <version>${tika.version}</version>
+ <version>{tika-version}</version>
</dependency>
----
@@ -196,7 +180,7 @@ value, clipped to whatever remains of the task's total budget.
|Minimum input file size in bytes. Smaller files are skipped.
|`maxFileSizeToOcr`
-|`2147483647` (~2 GB)
+|`52428800` (50 MB)
|Maximum input file size in bytes. Larger files are skipped.
|`maxImagePixels`
@@ -211,8 +195,8 @@ value, clipped to whatever remains of the task's total budget.
== Recommended: Docker + tika-pipes
Because this parser loads native code into the JVM, the safest deployment
-is a Docker container running tika-pipes with forked child processes.
-If the native code crashes, only the child process dies -- tika-pipes will
+is a Docker container running tika-pipes with forked JVMs.
+If the native code crashes, only the fork dies -- tika-pipes will
respawn it automatically.
A minimal `Dockerfile`:
@@ -236,21 +220,7 @@ WORKDIR /app
ENTRYPOINT ["java", "-jar", "tika-pipes-app.jar"]
----
-With the following parser configuration:
-
-[source,json]
-----
-{
- "parsers": [
- {
- "name": "tess4j-parser",
- "dataPath": "/usr/share/tesseract-ocr/5/tessdata",
- "nativeLibPath": "/usr/lib/x86_64-linux-gnu",
- "poolSize": 4
- }
- ]
-}
-----
+Use the <<_basic_configuration,basic configuration>> above with those container paths.
TIP: Set `poolSize` equal to the number of forked parser threads to
maximize throughput without over-allocating native memory.
@@ -288,8 +258,7 @@ maximize throughput without over-allocating native memory.
== Per-request configuration
-Override configuration for a single parse call by placing a `Tess4JConfig`
-on the `ParseContext`:
+In-process, place a `Tess4JConfig` on the `ParseContext`:
[source,java]
----
@@ -301,8 +270,7 @@ ParseContext context = new ParseContext();
context.set(Tess4JConfig.class, override);
----
-Note: `dataPath` and `nativeLibPath` cannot be changed at parse time
-(they are locked at parser initialization). Attempting to set them in a
-runtime config will throw `TikaConfigException`.
-
-@since Apache Tika 4.0
+NOTE: In caller-supplied JSON config — the tika-server `/config` endpoints, tika-grpc's
+`parse_context_json` — `dataPath` and `nativeLibPath` are locked at parser initialization, and
+setting either throws `TikaConfigException`. They point at native code, so a caller must not be
+able to move them.
diff --git a/docs/modules/ROOT/pages/configuration/parsers/tesseract-ocr-parser.adoc b/docs/modules/ROOT/pages/configuration/parsers/tesseract-ocr-parser.adoc
index 5cd5188196..73f93b0fcc 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/tesseract-ocr-parser.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/tesseract-ocr-parser.adoc
@@ -17,7 +17,8 @@
= TesseractOCRParser Configuration
-This page documents the configuration options for `TesseractOCRParser` in Tika 4.x.
+Configuration options for `TesseractOCRParser`, which runs the `tesseract` command-line program in
+a separate process.
== Basic Configuration
@@ -29,10 +30,10 @@ icon:github[] https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers
== Full Configuration
-The example below lists every option with its default value and an inline comment describing
-it. It also includes a `default-parser` entry so the config works as-is; see
-xref:configuration/index.adoc[Configuration] for why that entry matters. ImageMagick is
-optional — it is only used when `enableImagePreprocessing` or `applyRotation` is true.
+Every option with its default value and an inline comment describing it. The `default-parser` entry
+keeps the rest of Tika's parsers loaded — see xref:configuration/index.adoc[Configuration] for why
+that matters. ImageMagick is optional; it is used only when `enableImagePreprocessing` or
+`applyRotation` is true.
[source,json]
----
@@ -40,30 +41,7 @@ include::example$tesseract-full.json[]
----
icon:github[] https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/tesseract-full.json[View source on GitHub]
-== Changes from 3.x
-
-In Tika 3.x, the `otherTesseractSettings` was a list of space-delimited key-value strings:
-
-[source,xml]
-----
-<!-- 3.x XML format -->
-<param name="otherTesseractSettings" type="list">
- <string>textord_initialx_ile 0.75</string>
- <string>textord_noise_hfract 0.15625</string>
-</param>
-----
-
-In Tika 4.x, this is replaced with `otherTesseractConfig` as a proper map:
-
-[source,json]
-----
-// 4.x JSON format
-"otherTesseractConfig": {
- "textord_initialx_ile": "0.75",
- "textord_noise_hfract": "0.15625"
-}
-----
-
-The automatic converter handles this transformation.
-
-See xref:migration-to-4x/migrating-to-4x.adoc[Migrating to 4.x] for general migration guidance.
+Arbitrary Tesseract variables go in `otherTesseractConfig` as a name-to-value map, as in the
+example above. Converting a 3.x config? See
+xref:migration-to-4x/migrating-to-4x.adoc[Migrating to 4.x] — the automatic converter rewrites the
+old `otherTesseractSettings` list for you.
diff --git a/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc b/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
index d0b66e08e3..895c162529 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
@@ -22,11 +22,12 @@ to remote Vision-Language Model (VLM) endpoints. These parsers send images
(or PDFs) to an external API and convert the model's markdown response into
structured XHTML.
-Three implementations are provided out of the box:
+Three implementations are provided out of the box. Only `openai-vlm-parser` is auto-loaded via SPI;
+the other two must be named explicitly in your configuration.
[cols="1,2,1,1"]
|===
-|Parser |Endpoint |Config key |SPI auto-loaded?
+|Parser |Endpoint |Config key |Auto-loaded
|`OpenAIVLMParser`
|Any OpenAI-compatible chat completions endpoint (vLLM, Ollama, local FastAPI, OpenAI)
@@ -44,28 +45,18 @@ Three implementations are provided out of the box:
|No
|===
-`OpenAIVLMParser` is the only parser loaded by the default parser via SPI.
-`ClaudeVLMParser` and `GeminiVLMParser` must be explicitly added to your
-configuration.
-
-== Supported input types
-
-All three parsers handle standard OCR image types (`image/ocr-png`,
-`image/ocr-jpeg`, etc.). `ClaudeVLMParser` and `GeminiVLMParser`
-additionally declare `application/pdf` support, meaning they can process
-PDFs natively using the model's vision capabilities.
+All three handle the standard OCR image types (`image/ocr-png`, `image/ocr-jpeg`, ...).
+`ClaudeVLMParser` and `GeminiVLMParser` additionally declare `application/pdf`, so they can process
+PDFs natively with the model's vision capabilities.
== Module dependency
-The VLM parsers live in the `tika-parser-vlm-ocr-module` artifact. Add it
-to your project:
-
-[source,xml]
+[source,xml,subs=attributes+]
----
<dependency>
<groupId>org.apache.tika</groupId>
- <artifactId>tika-parser-vlm-ocr-module</artifactId>
- <version>${tika.version}</version>
+ <artifactId>tika-vlm</artifactId>
+ <version>{tika-version}</version>
</dependency>
----
@@ -216,7 +207,10 @@ text into the parent document's content stream. Mirrors
|`allowRuntimePrompt`
|`false`
-|When `false` (default), the `prompt` is fixed at initialization time and per-request overrides are rejected. When `true`, the prompt can be overridden per request via the `ParseContext` `VLMOCRConfig`. Security-relevant: a runtime-controllable prompt is effectively a prompt-injection surface for any caller that can set the `ParseContext`. Only enable when callers are trusted.
+|Whether a caller-supplied JSON config may override `prompt`. `false` pins the prompt to its
+initialization-time value and rejects the override. Security-relevant: a runtime-controllable prompt
+is a prompt-injection surface for anyone who can send per-request config, so enable it only for
+trusted callers. See <<_per_request_configuration,Per-request configuration>>.
|`completionsPath`
|`/v1/chat/completions` (OpenAI/vLLM only)
@@ -230,10 +224,13 @@ https://github.com/commonmark/commonmark-java[commonmark-java] and emits
... 13712 lines suppressed ...