You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@creadur.apache.org by "dependabot[bot] (via GitHub)" <gi...@apache.org> on 2026/08/24 00:32:44 UTC
[PR] Bump tika.version from 3.3.2 to 4.0.0 [creadur-rat]
dependabot[bot] opened a new pull request, #714:
URL: https://github.com/apache/creadur-rat/pull/714
Bumps `tika.version` from 3.3.2 to 4.0.0.
Updates `org.apache.tika:tika-core` from 3.3.2 to 4.0.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/apache/tika/blob/main/CHANGES.txt">org.apache.tika:tika-core's changelog</a>.</em></p>
<blockquote>
<p>Release 4.1.0 - unreleased</p>
<ul>
<li>
<p>The Kafka pipes iterator no longer stops at the first empty poll. A newly
subscribed consumer spends its first poll(s) joining the group and returns
empty even when the topic has a backlog, so the iterator could enqueue zero
files and report success. It now waits for a partition assignment (bounded
by the new assignmentTimeoutMs, default 30s) and requires a continuous quiet
window (drainIdleMs, default 1s) before concluding the topic is drained.
groupInitialRebalanceDelayMs is deprecated and no longer sent to the
consumer: it is a broker setting that Kafka has always ignored (TIKA-4833).</p>
</li>
<li>
<p>Pipes IPC: carry inline document bytes as a raw binary field beside the
tuple in the request envelope -- never inside the tuple or its
ParseContext -- and disable Smile's 7-bit binary encoding. Tuple JSON
serialized by 4.0.0 with an "inline-bytes" parse-context entry no longer
loads; it is rejected with a tailored message (TIKA-4829).</p>
</li>
<li>
<p>Digesting embedded documents no longer buffers each embedded object to a
temp file. Zip entries are re-read from the parent archive on rewind, and
a new process-wide CacheMemoryBudget (seeded by the pipes forked server;
default 256MB, clamped to a quarter of the fork's heap; tunable via
-Dtika.pipes.cacheMemoryBudgetBytes in the config's forkedJvmArgs, <=0
disables) lets embedded objects stay in memory past the per-object 1MB
threshold. New public API on TikaInputStream: get(IOSupplier,...),
enableRewind(CacheMemoryBudget), getSeekableByteChannel(). Zip/7z/epub/odf
parsing and zip container detection now read through seekable channels, so
after detection/parsing a TikaInputStream may no longer be file-backed
(hasFile() false); getPath()/getFile() still work and spool on demand
(TIKA-4828).</p>
</li>
<li>
<p>Pipes now carries the caller-supplied Content-Type across the worker's
fresh-metadata boundary as a soft detection hint, so every forked-parse
endpoint (/tika, /meta, /rmeta, /unpack, /async, /pipes, plus tika-grpc
and embedded PipesForkParser) can route on a client Content-Type, not
only on the filename. Detection keeps the hint only when it equals or
specializes the content-detected type (e.g. refining image/tiff to
image/x-canon-cr2); for bytes with no magic it can select any type,
matching the routing power the filename already had. The
CONTENT_TYPE_USER_OVERRIDE key is deliberately not carried, so the hint
cannot force an unrelated type (TIKA-4825).</p>
</li>
<li>
<p>OneNote extraction now follows document order, omits superseded page
revisions, sorts author metadata, extracts embedded object BLOBs, and
bounds malformed-input recursion and file-derived allocations. Parse
warnings and embedded relationship IDs are exposed in metadata. Malformed
or truncated files that cannot be fully parsed, and files whose walk
yields no content, now fall back to the legacy string dump instead of
failing or returning empty output. The legacy MS-ONESTORE walker bounds
its recursion (depth caps plus file-node-list and fragment-chain cycle
guards) and now honors shouldParseEmbedded for embedded file data</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/apache/tika/commit/514e1b3d8d29726d02ac6a12479d95f5db263379"><code>514e1b3</code></a> [maven-release-plugin] prepare release 4.0.0-rc1</li>
<li><a href="https://github.com/apache/tika/commit/4e39e0743878ad8e0d5a4019eabc08ff76911c3e"><code>4e39e07</code></a> revert second rc1 attempt</li>
<li><a href="https://github.com/apache/tika/commit/7975986d2a4162e227bd120e386dc626602d09ce"><code>7975986</code></a> javadocs take 42</li>
<li><a href="https://github.com/apache/tika/commit/95d423592d75d4f5d9fa0a4b25fe24d35ecb647d"><code>95d4235</code></a> [maven-release-plugin] prepare for next development iteration</li>
<li><a href="https://github.com/apache/tika/commit/666289be6fd30ed3587329c8ca86ab883a5d36b2"><code>666289b</code></a> [maven-release-plugin] prepare release 4.0.0-rc1</li>
<li><a href="https://github.com/apache/tika/commit/c9f6585992b4dc686457015c6d9650f7d55623af"><code>c9f6585</code></a> TIKA-4808 - revert aborted 4.0.0-rc1 release commits; fix per-module javadoc ...</li>
<li><a href="https://github.com/apache/tika/commit/41183e76e87862dc64468241abcd924569a11a1a"><code>41183e7</code></a> [maven-release-plugin] prepare for next development iteration</li>
<li><a href="https://github.com/apache/tika/commit/5dd7fc74ee20630375f44daddd21efcd3a5df64b"><code>5dd7fc7</code></a> [maven-release-plugin] prepare release 4.0.0-rc1</li>
<li><a href="https://github.com/apache/tika/commit/4b231cf01f622b62c4ff272787e993593b23a6aa"><code>4b231cf</code></a> TIKA-4808 -- prep CHANGES.txt for release</li>
<li><a href="https://github.com/apache/tika/commit/532a685ea1bd4db96befd17447f3a9aec70bca54"><code>532a685</code></a> TIKA-4808 - remove access to the network parser from cli (<a href="https://redirect.github.com/apache/tika/issues/3036">#3036</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/apache/tika/compare/3.3.2...4.0.0">compare view</a></li>
</ul>
</details>
<br />
Updates `org.apache.tika:tika-parser-text-module` from 3.3.2 to 4.0.0
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw merged PR #714:
URL: https://github.com/apache/creadur-rat/pull/714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5635760194
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_creadur-rat&pullRequest=714&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_creadur-rat&pullRequest=714&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_creadur-rat&pullRequest=714&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5666252357
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_creadur-rat&pullRequest=714&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_creadur-rat&pullRequest=714&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_creadur-rat&pullRequest=714&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3852102620
##########
src/changes/changes.xml:
##########
@@ -68,6 +68,9 @@ in order to be properly linked in site reports.
</release>
-->
<release version="1.0.0-SNAPSHOT" date="xxxx-yy-zz" description="Current SNAPSHOT - release to be done">
+ <action issue="RAT-532" type="add" dev="pottlinger">
+ Update to Tika 4.0.0: new charset detection logic in Tika returns different values compared to 3.x before, such as windows-1252 instead of ISO-8859-1.
Review Comment:
I think this is an error. We have some windows-1252 files but most are ISO-8859-1
I think for our purpose we can label windows-1252 as ISO-8859-1. I need to check the list that is returned from the new Tika and see if it includes ISO-8859-1 as one of the encodings. I think we should select ISO over windows when we have the option. This PR needs investigation and work.
##########
apache-rat-core/src/main/java/org/apache/rat/analysis/TikaProcessor.java:
##########
@@ -166,20 +177,30 @@ public static String process(final Document document) throws RatDocumentAnalysis
* @throws UnsupportedCharsetException on unsupported charset.
*/
private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException {
- final int bytesForCharsetDetection = 256;
- CharsetDetector encodingDetector = new CharsetDetector(bytesForCharsetDetection);
- encodingDetector.setText(stream);
- CharsetMatch charsetMatch = encodingDetector.detect();
- if (charsetMatch != null) {
- try {
- return Charset.forName(charsetMatch.getName());
- } catch (UnsupportedCharsetException e) {
- DefaultLog.getInstance().warn(String.format("Unsupported character set '%s' in file '%s'",
- charsetMatch.getName(), documentName));
- throw e;
+ stream.mark(BYTES_FOR_CHARSET_DETECTION);
+ try {
+ byte[] sample = stream.readNBytes(BYTES_FOR_CHARSET_DETECTION);
+ if (sample.length == 0) {
+ DefaultLog.getInstance().debug(String.format("No contents in file '%s'", documentName));
+ return null;
+ }
+
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+
+ try (TikaInputStream tis = TikaInputStream.get(sample, metadata)) {
+ List<EncodingResult> results = ENCODING_DETECTOR.detect(tis, metadata, parseContext);
+
+ if (results.isEmpty()) {
+ DefaultLog.getInstance().debug(String.format("No encoding found for file '%s'", documentName));
+ return null;
+ }
Review Comment:
This code does not do the same thing. the debug should be a warning.
And what happend to unsupported character sets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3852205005
##########
src/changes/changes.xml:
##########
@@ -68,6 +68,9 @@ in order to be properly linked in site reports.
</release>
-->
<release version="1.0.0-SNAPSHOT" date="xxxx-yy-zz" description="Current SNAPSHOT - release to be done">
+ <action issue="RAT-532" type="add" dev="pottlinger">
+ Update to Tika 4.0.0: new charset detection logic in Tika returns different values compared to 3.x before, such as windows-1252 instead of ISO-8859-1.
Review Comment:
The Tika4 logics is to return the "best" charset. In contrast to version 3.x this changed into windows-1252. The new implementation returns the first hit. Personally I wouldn't want to introduce new logics on the RAT-side to generalise into ISO-8859-1 and would take the change as tika-induced and document it in our changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3884267850
##########
apache-rat-core/src/main/java/org/apache/rat/analysis/TikaProcessor.java:
##########
@@ -166,20 +177,30 @@ public static String process(final Document document) throws RatDocumentAnalysis
* @throws UnsupportedCharsetException on unsupported charset.
*/
private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException {
- final int bytesForCharsetDetection = 256;
- CharsetDetector encodingDetector = new CharsetDetector(bytesForCharsetDetection);
- encodingDetector.setText(stream);
- CharsetMatch charsetMatch = encodingDetector.detect();
- if (charsetMatch != null) {
- try {
- return Charset.forName(charsetMatch.getName());
- } catch (UnsupportedCharsetException e) {
- DefaultLog.getInstance().warn(String.format("Unsupported character set '%s' in file '%s'",
- charsetMatch.getName(), documentName));
- throw e;
+ stream.mark(BYTES_FOR_CHARSET_DETECTION);
+ try {
+ byte[] sample = stream.readNBytes(BYTES_FOR_CHARSET_DETECTION);
+ if (sample.length == 0) {
+ DefaultLog.getInstance().debug(String.format("No contents in file '%s'", documentName));
+ return null;
+ }
+
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+
+ try (TikaInputStream tis = TikaInputStream.get(sample, metadata)) {
+ List<EncodingResult> results = ENCODING_DETECTOR.detect(tis, metadata, parseContext);
+
+ if (results.isEmpty()) {
+ DefaultLog.getInstance().debug(String.format("No encoding found for file '%s'", documentName));
+ return null;
+ }
Review Comment:
I tried adding "random bytes" but Tika still reports a probabilistic value and I'm unable to provide an input that yields an empty result in order to test RAT's behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5458285751
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate failed**
Failed conditions
 [66.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list) (required ≥ 80%)
<!-- sqra:qg_revision=c2eab3946c51972c579343cd2e5491f716281d3d8de3aad81561cf7b4be57b3c -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5401857648
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate failed**
Failed conditions
 [66.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list) (required ≥ 80%)
<!-- sqra:qg_revision=c2eab3946c51972c579343cd2e5491f716281d3d8de3aad81561cf7b4be57b3c -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5436305178
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate failed**
Failed conditions
 [66.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list) (required ≥ 80%)
<!-- sqra:qg_revision=c2eab3946c51972c579343cd2e5491f716281d3d8de3aad81561cf7b4be57b3c -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5634653977
## [](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714) **Quality Gate failed**
Failed conditions
 [66.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_creadur-rat&pullRequest=714&metric=new_coverage&view=list) (required ≥ 80%)
<!-- sqra:qg_revision=c2eab3946c51972c579343cd2e5491f716281d3d8de3aad81561cf7b4be57b3c -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_creadur-rat&pullRequest=714)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5635882770
@Claudenw I've added unit tests to raise the coverage. Would you mind taking a look? As the new tika works differently I think we should adapt and add this in the changelog. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3852213520
##########
apache-rat-core/src/main/java/org/apache/rat/analysis/TikaProcessor.java:
##########
@@ -166,20 +177,30 @@ public static String process(final Document document) throws RatDocumentAnalysis
* @throws UnsupportedCharsetException on unsupported charset.
*/
private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException {
- final int bytesForCharsetDetection = 256;
- CharsetDetector encodingDetector = new CharsetDetector(bytesForCharsetDetection);
- encodingDetector.setText(stream);
- CharsetMatch charsetMatch = encodingDetector.detect();
- if (charsetMatch != null) {
- try {
- return Charset.forName(charsetMatch.getName());
- } catch (UnsupportedCharsetException e) {
- DefaultLog.getInstance().warn(String.format("Unsupported character set '%s' in file '%s'",
- charsetMatch.getName(), documentName));
- throw e;
+ stream.mark(BYTES_FOR_CHARSET_DETECTION);
+ try {
+ byte[] sample = stream.readNBytes(BYTES_FOR_CHARSET_DETECTION);
+ if (sample.length == 0) {
+ DefaultLog.getInstance().debug(String.format("No contents in file '%s'", documentName));
+ return null;
+ }
+
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+
+ try (TikaInputStream tis = TikaInputStream.get(sample, metadata)) {
+ List<EncodingResult> results = ENCODING_DETECTOR.detect(tis, metadata, parseContext);
+
+ if (results.isEmpty()) {
+ DefaultLog.getInstance().debug(String.format("No encoding found for file '%s'", documentName));
+ return null;
+ }
Review Comment:
We do not have an explicit test for unsuppoorted character sets. Tika handles this internally and returns no charset. If no charset is returned RAT will mark as UNKNOWN if I'm not too mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#issuecomment-5401897302
@Claudenw would you mind starting a review (I'll add the tests to pass the quality build later). The new Tika4 API detects charsets differently, thus so many changes in test expectations. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3860617969
##########
src/changes/changes.xml:
##########
@@ -68,6 +68,9 @@ in order to be properly linked in site reports.
</release>
-->
<release version="1.0.0-SNAPSHOT" date="xxxx-yy-zz" description="Current SNAPSHOT - release to be done">
+ <action issue="RAT-532" type="add" dev="pottlinger">
+ Update to Tika 4.0.0: new charset detection logic in Tika returns different values compared to 3.x before, such as windows-1252 instead of ISO-8859-1.
Review Comment:
Out of curiosity I added some logging locally in TikaProcessor to just list the encodings:
```
[INFO] Running org.apache.rat.analysis.AnalyserFactoryTest
INFO: >>> /jira/RAT147/windows-newlines.txt.bin has encodings: [UTF-8@1.00[DECLARATIVE]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /jira/RAT147/unix-newlines.txt.bin has encodings: [UTF-8@1.00[DECLARATIVE]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/LICENSE has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt has encodings: [windows-1252@0.10[STATISTICAL]]
....
[INFO] Running org.apache.rat.report.xml.XmlReportFactoryTest
INFO: >>> /ILoggerFactory.java has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /LICENSE has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /NOTICE has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Source.java has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Text.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /TextHttps.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /Xml.xml has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /buildr.rb has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /generated.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /sub/Empty.txt has encodings: [windows-1252@0.10[STATISTICAL]]
INFO: >>> /tri.txt has encodings: [windows-1252@0.10[STATISTICAL]]
```
It seems that ISO is not recognized anymore with Tika 4.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-532: Bump tika.version from 3.3.2 to 4.0.0 and migrate charset detection [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3852213520
##########
apache-rat-core/src/main/java/org/apache/rat/analysis/TikaProcessor.java:
##########
@@ -166,20 +177,30 @@ public static String process(final Document document) throws RatDocumentAnalysis
* @throws UnsupportedCharsetException on unsupported charset.
*/
private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException {
- final int bytesForCharsetDetection = 256;
- CharsetDetector encodingDetector = new CharsetDetector(bytesForCharsetDetection);
- encodingDetector.setText(stream);
- CharsetMatch charsetMatch = encodingDetector.detect();
- if (charsetMatch != null) {
- try {
- return Charset.forName(charsetMatch.getName());
- } catch (UnsupportedCharsetException e) {
- DefaultLog.getInstance().warn(String.format("Unsupported character set '%s' in file '%s'",
- charsetMatch.getName(), documentName));
- throw e;
+ stream.mark(BYTES_FOR_CHARSET_DETECTION);
+ try {
+ byte[] sample = stream.readNBytes(BYTES_FOR_CHARSET_DETECTION);
+ if (sample.length == 0) {
+ DefaultLog.getInstance().debug(String.format("No contents in file '%s'", documentName));
+ return null;
+ }
+
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+
+ try (TikaInputStream tis = TikaInputStream.get(sample, metadata)) {
+ List<EncodingResult> results = ENCODING_DETECTOR.detect(tis, metadata, parseContext);
+
+ if (results.isEmpty()) {
+ DefaultLog.getInstance().debug(String.format("No encoding found for file '%s'", documentName));
+ return null;
+ }
Review Comment:
We do not have an explicit test for unsupported character sets. Tika handles this internally and returns no charset. If no charset is returned RAT will mark as UNKNOWN if I'm not too mistaken.
Do you have an example file that triggers this exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org