You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by dp...@apache.org on 2026/09/25 18:29:46 UTC
(stormcrawler) branch tika-pipes-parser-timeout created (now 99b2032d)
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a change to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
at 99b2032d Reformat
This branch includes the following new commits:
new 985865cc #2097 ParserBolt: default the forked JVM to -Xmx512m
new 38775a9f #2097 ParserBolt: start the fork in prepare
new afbdbb65 #2097 ParserBolt: document the fork sizing and defaults
new fdfa246e #2097 ParserBolt: leave the fork's socket timeout to Tika
new 99b2032d Reformat
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(stormcrawler) 01/05: #2097 ParserBolt: default the forked JVM to -Xmx512m
Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 985865ccf992b4a71a569cc381c62910a76d11ff
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 19:25:41 2026 +0200
#2097 ParserBolt: default the forked JVM to -Xmx512m
Without a heap flag Tika gives each fork 60% of the RAM, whatever the
number of ParserBolt executors on the host.
Signed-off-by: Davide Polato <dp...@apache.org>
---
docs/src/main/asciidoc/configuration.adoc | 2 +-
external/tika/README.md | 2 +-
.../org/apache/stormcrawler/tika/ParserBolt.java | 33 ++++++++++--
.../tika/ParserBoltPipesJvmArgsTest.java | 63 ++++++++++++++++++++++
4 files changed, 93 insertions(+), 7 deletions(-)
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index 536ef8a4..76fba99e 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -571,7 +571,7 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/tika[tika
| parser.extract.embedded | false | Whether to parse embedded documents. Since Tika 4 embedded documents are no longer parsed unless this is set to `true`.
| parser.tika.text.maxlength | -1 | Maximum number of characters of text extracted from a document, -1 (or any negative value) for no limit. When the limit is reached the parse stops, the text and outlinks extracted so far are kept and the metadata `parse.text.trimmed` is set to `true`.
| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. Keep the value below `topology.message.timeout.secs`. `parser.htmlmapper.classname` is ignored in this mode, see the README of the module.
-| parser.tika.pipes.jvmargs | - | JVM arguments passed to each forked process under `parser.tika.timeout`, e.g. `-Xmx512m`.
+| parser.tika.pipes.jvmargs | -Xmx512m | JVM arguments passed to each forked process under `parser.tika.timeout`. `-Xmx512m` is added when they set no maximum heap (`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxRAM*`).
| parser.tika.pipes.maxfilesperprocess | - | Restart a forked process after this many documents under `parser.tika.timeout`, to bound slow leaks in parsing libraries.
| parser.tika.pipes.plugins.dir | - | Directory holding Tika Pipes plugin zips under `parser.tika.timeout`, not needed by default. Unset uses Tika's default plugin directory resolution.
|===
diff --git a/external/tika/README.md b/external/tika/README.md
index d1a77e44..fd0bb306 100644
--- a/external/tika/README.md
+++ b/external/tika/README.md
@@ -41,7 +41,7 @@ The length of the text extracted from a document can be limited with `parser.tik
The time spent parsing a document can be limited with `parser.tika.timeout` (milliseconds, default `-1`, 0 or less means no limit). When set, the parse runs in a forked JVM via [Tika Pipes](https://tika.apache.org/docs/4.0.x/pipes/index.html), one per bolt instance, started with the same `java` and classpath as the Storm worker: a document that takes longer than the timeout is killed outright (not merely asked to stop) and sent to the status stream as an `ERROR` with the message `parse t [...]
-Each forked JVM needs memory on top of the worker, so size the hosts for the number of `ParserBolt` executors. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (e.g. `-Xmx512m`), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked JVM through a temporary file in the [...]
+Each forked JVM needs memory on top of the worker, so size the hosts for the number of `ParserBolt` executors. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (`-Xmx512m` is added when they set no maximum heap), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked J [...]
`parser.htmlmapper.classname` is not applied to parses running under `parser.tika.timeout`: an `HtmlMapper` cannot be passed to the forked JVM, which always uses Tika's `DefaultHtmlMapper`. That mapper drops the elements it does not consider safe from the DOM given to the parse filters, so XPath expressions written against the default `IdentityHtmlMapper` output may need adjusting.
diff --git a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
index c3e3cf74..db6e9292 100644
--- a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
+++ b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
@@ -126,9 +126,15 @@ public class ParserBolt extends BaseRichBolt {
*/
public static final String PIPES_PLUGINS_DIR_PARAM = "parser.tika.pipes.plugins.dir";
- /** JVM arguments passed to each forked process under {@link #PARSE_TIMEOUT_PARAM}. */
+ /**
+ * JVM arguments passed to each forked process under {@link #PARSE_TIMEOUT_PARAM}. The fork gets
+ * {@code -Xmx512m} when they set no maximum heap.
+ */
public static final String PIPES_JVM_ARGS_PARAM = "parser.tika.pipes.jvmargs";
+ /** Heap of a forked process when {@link #PIPES_JVM_ARGS_PARAM} sets none. */
+ private static final String PIPES_DEFAULT_HEAP = "-Xmx512m";
+
/** Restart a forked process after this many documents under {@link #PARSE_TIMEOUT_PARAM}. */
public static final String PIPES_MAX_FILES_PER_PROCESS_PARAM =
"parser.tika.pipes.maxfilesperprocess";
@@ -575,10 +581,7 @@ public class ParserBolt extends BaseRichBolt {
if (StringUtils.isNotBlank(pluginsDir)) {
pipesConfig.setPluginsDir(Paths.get(pluginsDir));
}
- List<String> jvmArgs = ConfUtils.loadListFromConf(PIPES_JVM_ARGS_PARAM, conf);
- if (!jvmArgs.isEmpty()) {
- pipesConfig.setJvmArgs(jvmArgs);
- }
+ pipesConfig.setJvmArgs(forkedJvmArgs(conf));
// an HtmlMapper can't be passed to the forked JVM, which always uses Tika's default
if (!DefaultHtmlMapper.class.equals(htmlMapperClass)) {
@@ -600,6 +603,26 @@ public class ParserBolt extends BaseRichBolt {
}
}
+ /**
+ * Returns the JVM arguments of a forked process: the configured ones, plus {@link
+ * #PIPES_DEFAULT_HEAP} when none of them sets a maximum heap. Without it Tika gives each fork 60% of
+ * the RAM, whatever the number of forks on the host.
+ */
+ static List<String> forkedJvmArgs(Map<String, Object> conf) {
+ final List<String> jvmArgs =
+ new ArrayList<>(ConfUtils.loadListFromConf(PIPES_JVM_ARGS_PARAM, conf));
+ if (jvmArgs.stream().noneMatch(ParserBolt::setsHeap)) {
+ jvmArgs.add(PIPES_DEFAULT_HEAP);
+ }
+ return jvmArgs;
+ }
+
+ private static boolean setsHeap(String jvmArg) {
+ return jvmArg.startsWith("-Xmx")
+ || jvmArg.startsWith("-XX:MaxHeapSize")
+ || jvmArg.startsWith("-XX:MaxRAM");
+ }
+
/**
* The parse under {@link #PARSE_TIMEOUT_PARAM} did not complete within {@link #parseTimeout}.
*/
diff --git a/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java
new file mode 100644
index 00000000..1478b200
--- /dev/null
+++ b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.stormcrawler.tika;
+
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class ParserBoltPipesJvmArgsTest {
+
+ @Test
+ void defaultHeapWhenNoArgumentIsSet() {
+ Assertions.assertEquals(List.of("-Xmx512m"), ParserBolt.forkedJvmArgs(Map.of()));
+ }
+
+ @Test
+ void defaultHeapIsAddedToOtherArguments() {
+ Assertions.assertEquals(
+ List.of("-Djava.awt.headless=true", "-Xmx512m"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-Djava.awt.headless=true")));
+ }
+
+ @Test
+ void configuredHeapIsKept() {
+ Assertions.assertEquals(
+ List.of("-Xmx1g"),
+ ParserBolt.forkedJvmArgs(Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-Xmx1g")));
+ }
+
+ @Test
+ void configuredMaxHeapSizeIsKept() {
+ Assertions.assertEquals(
+ List.of("-XX:MaxHeapSize=2g"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-XX:MaxHeapSize=2g")));
+ }
+
+ @Test
+ void configuredRamPercentageIsKept() {
+ Assertions.assertEquals(
+ List.of("-XX:MaxRAMPercentage=25"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(
+ ParserBolt.PIPES_JVM_ARGS_PARAM,
+ List.of("-XX:MaxRAMPercentage=25"))));
+ }
+}
(stormcrawler) 03/05: #2097 ParserBolt: document the fork sizing and defaults
Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit afbdbb65062881d73fd35418c3e7c187d8ab7b4c
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 19:25:47 2026 +0200
#2097 ParserBolt: document the fork sizing and defaults
Signed-off-by: Davide Polato <dp...@apache.org>
---
docs/src/main/asciidoc/configuration.adoc | 4 ++--
external/tika/README.md | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index 76fba99e..14887660 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -570,9 +570,9 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/tika[tika
| parser.tika.config.file | tika-config.json | Name of the classpath resource holding the Tika configuration (JSON format since Tika 4).
| parser.extract.embedded | false | Whether to parse embedded documents. Since Tika 4 embedded documents are no longer parsed unless this is set to `true`.
| parser.tika.text.maxlength | -1 | Maximum number of characters of text extracted from a document, -1 (or any negative value) for no limit. When the limit is reached the parse stops, the text and outlinks extracted so far are kept and the metadata `parse.text.trimmed` is set to `true`.
-| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. Keep the value below `topology.message.timeout.secs`. `parser.htmlmapper.classname` is ignored in this mode, see the README of the module.
+| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. A partial result returned between embedded documents is kept and marked `parse.text.trimmed`. Keep it well below `topology.message.timeout.secs`: the time a tuple waits behind [...]
| parser.tika.pipes.jvmargs | -Xmx512m | JVM arguments passed to each forked process under `parser.tika.timeout`. `-Xmx512m` is added when they set no maximum heap (`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxRAM*`).
-| parser.tika.pipes.maxfilesperprocess | - | Restart a forked process after this many documents under `parser.tika.timeout`, to bound slow leaks in parsing libraries.
+| parser.tika.pipes.maxfilesperprocess | - | Restart a forked process after this many documents under `parser.tika.timeout`, to bound slow leaks in parsing libraries. Unset uses Tika's default, 10000.
| parser.tika.pipes.plugins.dir | - | Directory holding Tika Pipes plugin zips under `parser.tika.timeout`, not needed by default. Unset uses Tika's default plugin directory resolution.
|===
diff --git a/external/tika/README.md b/external/tika/README.md
index fd0bb306..ac7ef37a 100644
--- a/external/tika/README.md
+++ b/external/tika/README.md
@@ -39,9 +39,9 @@ Embedded documents are only parsed when `parser.extract.embedded` is set to `tru
The length of the text extracted from a document can be limited with `parser.tika.text.maxlength` (number of characters, default `-1`, any negative value means no limit). When the limit is reached the parse stops, the text and outlinks extracted so far are kept and the document is emitted with the metadata `parse.text.trimmed` set to `true`.
-The time spent parsing a document can be limited with `parser.tika.timeout` (milliseconds, default `-1`, 0 or less means no limit). When set, the parse runs in a forked JVM via [Tika Pipes](https://tika.apache.org/docs/4.0.x/pipes/index.html), one per bolt instance, started with the same `java` and classpath as the Storm worker: a document that takes longer than the timeout is killed outright (not merely asked to stop) and sent to the status stream as an `ERROR` with the message `parse t [...]
+The time spent parsing a document can be limited with `parser.tika.timeout` (milliseconds, default `-1`, 0 or less means no limit). When set, the parse runs in a forked JVM via [Tika Pipes](https://tika.apache.org/docs/4.0.x/pipes/index.html), one per bolt instance. ParserBolt starts the fork during `prepare()` using a small test parse. A parse that reaches the hard deadline is reported as `parse timeout`; a partial result returned between embedded documents is kept and marked `parse.tex [...]
-Each forked JVM needs memory on top of the worker, so size the hosts for the number of `ParserBolt` executors. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (`-Xmx512m` is added when they set no maximum heap), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked J [...]
+Each `ParserBolt` instance runs one fork, so a host needs the worker heap plus one fork heap per `ParserBolt` instance on it, and some overhead for each JVM. When several forks share a host, consider `-XX:ActiveProcessorCount` as well as a heap limit. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (`-Xmx512m` is added when they set no maximum heap), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsin [...]
`parser.htmlmapper.classname` is not applied to parses running under `parser.tika.timeout`: an `HtmlMapper` cannot be passed to the forked JVM, which always uses Tika's `DefaultHtmlMapper`. That mapper drops the elements it does not consider safe from the DOM given to the parse filters, so XPath expressions written against the default `IdentityHtmlMapper` output may need adjusting.
(stormcrawler) 04/05: #2097 ParserBolt: leave the fork's socket timeout to Tika
Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit fdfa246eb4a38e47eec180296dd6b3ba40faf76a
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 20:22:12 2026 +0200
#2097 ParserBolt: leave the fork's socket timeout to Tika
TimeoutLimits bounds the parse. From Tika 4.1 the socket timeout also
shuts down an idle fork and must stay above the 1s heartbeat.
Signed-off-by: Davide Polato <dp...@apache.org>
---
.../tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
index 538cda02..c122457c 100644
--- a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
+++ b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
@@ -549,8 +549,8 @@ public class ParserBolt extends BaseRichBolt {
}
/**
- * Builds the {@link PipesForkParser} used under {@link #PARSE_TIMEOUT_PARAM}: the parent kills
- * the forked process outright via {@code socketTimeoutMillis}, not cooperative interruption.
+ * Builds the {@link PipesForkParser} used under {@link #PARSE_TIMEOUT_PARAM}: the fork kills a parse
+ * that runs past its {@code TimeoutLimits} outright, without relying on cooperative interruption.
*/
private PipesForkParser buildPipesForkParser(Map<String, Object> conf) {
PipesForkParserConfig pipesConfig = new PipesForkParserConfig();
@@ -563,8 +563,6 @@ public class ParserBolt extends BaseRichBolt {
pipesConfig.setParseMode(ParseMode.CONCATENATE);
pipesConfig.setMaxEmbeddedCount(extractEmbedded ? -1 : 0);
pipesConfig.setTimeoutLimits(new TimeoutLimits(parseTimeout, parseTimeout));
- // the real enforcement: kills the forked process outright if it doesn't respond in time
- pipesConfig.getPipesConfig().setSocketTimeoutMillis(parseTimeout);
if (resolvedTikaConfigPath != null) {
pipesConfig.setUserConfigPath(resolvedTikaConfigPath);
}
(stormcrawler) 02/05: #2097 ParserBolt: start the fork in prepare
Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 38775a9f90b078ff8f7b6af703f9216601d8dd20
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 19:25:44 2026 +0200
#2097 ParserBolt: start the fork in prepare
A fork that cannot start, e.g. with a bad jvmarg, now fails the bolt
instead of turning every URL into a parse pipes error.
Signed-off-by: Davide Polato <dp...@apache.org>
---
.../org/apache/stormcrawler/tika/ParserBolt.java | 39 +++++++++++++++++++---
.../tika/ParserBoltPipesTimeoutTest.java | 9 +++++
2 files changed, 44 insertions(+), 4 deletions(-)
diff --git a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
index db6e9292..538cda02 100644
--- a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
+++ b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
@@ -25,6 +25,7 @@ import java.io.StringReader;
import java.net.MalformedURLException;
import java.net.URISyntaxException;
import java.net.URL;
+import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
@@ -253,6 +254,7 @@ public class ParserBolt extends BaseRichBolt {
if (parseTimeout > 0) {
pipesForkParser = buildPipesForkParser(conf);
+ startFork();
} else {
deleteTemporaryTikaConfig();
}
@@ -623,6 +625,37 @@ public class ParserBolt extends BaseRichBolt {
|| jvmArg.startsWith("-XX:MaxRAM");
}
+ /**
+ * Parses a one-word document so that a fork which cannot start fails the bolt here, instead of
+ * failing every document with a "parse pipes error" status. The fork starts lazily otherwise.
+ */
+ private void startFork() {
+ final PipesForkResult result;
+ try (TikaInputStream tis =
+ TikaInputStream.get("StormCrawler".getBytes(StandardCharsets.UTF_8))) {
+ result = pipesForkParser.parse(tis);
+ } catch (IOException | TikaException | PipesException e) {
+ throw new IllegalStateException(
+ "The Tika Pipes fork for " + PARSE_TIMEOUT_PARAM + " did not start", e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(
+ "Interrupted while starting the Tika Pipes fork for " + PARSE_TIMEOUT_PARAM, e);
+ }
+ if (!result.isSuccess()) {
+ throw new IllegalStateException(
+ "The Tika Pipes fork for "
+ + PARSE_TIMEOUT_PARAM
+ + " failed a test parse: "
+ + describe(result));
+ }
+ }
+
+ private static String describe(PipesForkResult result) {
+ return result.getStatus()
+ + (result.getMessage() != null ? " - " + result.getMessage() : "");
+ }
+
/**
* The parse under {@link #PARSE_TIMEOUT_PARAM} did not complete within {@link #parseTimeout}.
*/
@@ -684,8 +717,7 @@ public class ParserBolt extends BaseRichBolt {
"Tika parse of "
+ url
+ " crashed the forked process: "
- + result.getStatus()
- + (result.getMessage() != null ? " - " + result.getMessage() : ""));
+ + describe(result));
}
if (!result.isSuccess()) {
@@ -693,8 +725,7 @@ public class ParserBolt extends BaseRichBolt {
"Tika Pipes parse of "
+ url
+ " failed: "
- + result.getStatus()
- + (result.getMessage() != null ? " - " + result.getMessage() : ""));
+ + describe(result));
}
org.apache.tika.metadata.Metadata resultMetadata = result.getMetadata();
diff --git a/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesTimeoutTest.java b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesTimeoutTest.java
index 2cfb870f..0ce52b5b 100644
--- a/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesTimeoutTest.java
+++ b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesTimeoutTest.java
@@ -159,6 +159,15 @@ class ParserBoltPipesTimeoutTest extends ParsingTester {
Assertions.assertEquals("true", parseMetadata.getFirstValue(ParserBolt.TEXT_TRIMMED_KEY));
}
+ /** A fork which cannot start fails the bolt in prepare, not every document afterwards. */
+ @Test
+ @Timeout(60)
+ void forkThatCannotStartFailsPrepare() {
+ Map<String, Object> conf = new HashMap<>();
+ conf.put(ParserBolt.PIPES_JVM_ARGS_PARAM, "-XX:NoSuchOption");
+ Assertions.assertThrows(IllegalStateException.class, () -> prepare(conf));
+ }
+
/** A forked JVM dying mid-parse is reported as "parse crash" and the bolt carries on. */
@Test
@Timeout(60)
(stormcrawler) 05/05: Reformat
Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 99b2032d16616260e555e66e76e804334197516d
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 20:29:28 2026 +0200
Reformat
---
.../org/apache/stormcrawler/tika/ParserBolt.java | 20 +++++++-------------
1 file changed, 7 insertions(+), 13 deletions(-)
diff --git a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
index c122457c..450b1748 100644
--- a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
+++ b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
@@ -549,8 +549,9 @@ public class ParserBolt extends BaseRichBolt {
}
/**
- * Builds the {@link PipesForkParser} used under {@link #PARSE_TIMEOUT_PARAM}: the fork kills a parse
- * that runs past its {@code TimeoutLimits} outright, without relying on cooperative interruption.
+ * Builds the {@link PipesForkParser} used under {@link #PARSE_TIMEOUT_PARAM}: the fork kills a
+ * parse that runs past its {@code TimeoutLimits} outright, without relying on cooperative
+ * interruption.
*/
private PipesForkParser buildPipesForkParser(Map<String, Object> conf) {
PipesForkParserConfig pipesConfig = new PipesForkParserConfig();
@@ -605,8 +606,8 @@ public class ParserBolt extends BaseRichBolt {
/**
* Returns the JVM arguments of a forked process: the configured ones, plus {@link
- * #PIPES_DEFAULT_HEAP} when none of them sets a maximum heap. Without it Tika gives each fork 60% of
- * the RAM, whatever the number of forks on the host.
+ * #PIPES_DEFAULT_HEAP} when none of them sets a maximum heap. Without it Tika gives each fork
+ * 60% of the RAM, whatever the number of forks on the host.
*/
static List<String> forkedJvmArgs(Map<String, Object> conf) {
final List<String> jvmArgs =
@@ -712,18 +713,11 @@ public class ParserBolt extends BaseRichBolt {
"Tika parse of " + url + " exceeded " + parseTimeout + "ms");
}
throw new ParseCrashException(
- "Tika parse of "
- + url
- + " crashed the forked process: "
- + describe(result));
+ "Tika parse of " + url + " crashed the forked process: " + describe(result));
}
if (!result.isSuccess()) {
- throw new IOException(
- "Tika Pipes parse of "
- + url
- + " failed: "
- + describe(result));
+ throw new IOException("Tika Pipes parse of " + url + " failed: " + describe(result));
}
org.apache.tika.metadata.Metadata resultMetadata = result.getMetadata();