You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by dp...@apache.org on 2026/09/25 18:29:47 UTC
(stormcrawler) 01/05: #2097 ParserBolt: default the forked JVM to -Xmx512m
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch tika-pipes-parser-timeout
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 985865ccf992b4a71a569cc381c62910a76d11ff
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Sep 25 19:25:41 2026 +0200
#2097 ParserBolt: default the forked JVM to -Xmx512m
Without a heap flag Tika gives each fork 60% of the RAM, whatever the
number of ParserBolt executors on the host.
Signed-off-by: Davide Polato <dp...@apache.org>
---
docs/src/main/asciidoc/configuration.adoc | 2 +-
external/tika/README.md | 2 +-
.../org/apache/stormcrawler/tika/ParserBolt.java | 33 ++++++++++--
.../tika/ParserBoltPipesJvmArgsTest.java | 63 ++++++++++++++++++++++
4 files changed, 93 insertions(+), 7 deletions(-)
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index 536ef8a4..76fba99e 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -571,7 +571,7 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/tika[tika
| parser.extract.embedded | false | Whether to parse embedded documents. Since Tika 4 embedded documents are no longer parsed unless this is set to `true`.
| parser.tika.text.maxlength | -1 | Maximum number of characters of text extracted from a document, -1 (or any negative value) for no limit. When the limit is reached the parse stops, the text and outlinks extracted so far are kept and the metadata `parse.text.trimmed` is set to `true`.
| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. Keep the value below `topology.message.timeout.secs`. `parser.htmlmapper.classname` is ignored in this mode, see the README of the module.
-| parser.tika.pipes.jvmargs | - | JVM arguments passed to each forked process under `parser.tika.timeout`, e.g. `-Xmx512m`.
+| parser.tika.pipes.jvmargs | -Xmx512m | JVM arguments passed to each forked process under `parser.tika.timeout`. `-Xmx512m` is added when they set no maximum heap (`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxRAM*`).
| parser.tika.pipes.maxfilesperprocess | - | Restart a forked process after this many documents under `parser.tika.timeout`, to bound slow leaks in parsing libraries.
| parser.tika.pipes.plugins.dir | - | Directory holding Tika Pipes plugin zips under `parser.tika.timeout`, not needed by default. Unset uses Tika's default plugin directory resolution.
|===
diff --git a/external/tika/README.md b/external/tika/README.md
index d1a77e44..fd0bb306 100644
--- a/external/tika/README.md
+++ b/external/tika/README.md
@@ -41,7 +41,7 @@ The length of the text extracted from a document can be limited with `parser.tik
The time spent parsing a document can be limited with `parser.tika.timeout` (milliseconds, default `-1`, 0 or less means no limit). When set, the parse runs in a forked JVM via [Tika Pipes](https://tika.apache.org/docs/4.0.x/pipes/index.html), one per bolt instance, started with the same `java` and classpath as the Storm worker: a document that takes longer than the timeout is killed outright (not merely asked to stop) and sent to the status stream as an `ERROR` with the message `parse t [...]
-Each forked JVM needs memory on top of the worker, so size the hosts for the number of `ParserBolt` executors. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (e.g. `-Xmx512m`), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked JVM through a temporary file in the [...]
+Each forked JVM needs memory on top of the worker, so size the hosts for the number of `ParserBolt` executors. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (`-Xmx512m` is added when they set no maximum heap), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked J [...]
`parser.htmlmapper.classname` is not applied to parses running under `parser.tika.timeout`: an `HtmlMapper` cannot be passed to the forked JVM, which always uses Tika's `DefaultHtmlMapper`. That mapper drops the elements it does not consider safe from the DOM given to the parse filters, so XPath expressions written against the default `IdentityHtmlMapper` output may need adjusting.
diff --git a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
index c3e3cf74..db6e9292 100644
--- a/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
+++ b/external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java
@@ -126,9 +126,15 @@ public class ParserBolt extends BaseRichBolt {
*/
public static final String PIPES_PLUGINS_DIR_PARAM = "parser.tika.pipes.plugins.dir";
- /** JVM arguments passed to each forked process under {@link #PARSE_TIMEOUT_PARAM}. */
+ /**
+ * JVM arguments passed to each forked process under {@link #PARSE_TIMEOUT_PARAM}. The fork gets
+ * {@code -Xmx512m} when they set no maximum heap.
+ */
public static final String PIPES_JVM_ARGS_PARAM = "parser.tika.pipes.jvmargs";
+ /** Heap of a forked process when {@link #PIPES_JVM_ARGS_PARAM} sets none. */
+ private static final String PIPES_DEFAULT_HEAP = "-Xmx512m";
+
/** Restart a forked process after this many documents under {@link #PARSE_TIMEOUT_PARAM}. */
public static final String PIPES_MAX_FILES_PER_PROCESS_PARAM =
"parser.tika.pipes.maxfilesperprocess";
@@ -575,10 +581,7 @@ public class ParserBolt extends BaseRichBolt {
if (StringUtils.isNotBlank(pluginsDir)) {
pipesConfig.setPluginsDir(Paths.get(pluginsDir));
}
- List<String> jvmArgs = ConfUtils.loadListFromConf(PIPES_JVM_ARGS_PARAM, conf);
- if (!jvmArgs.isEmpty()) {
- pipesConfig.setJvmArgs(jvmArgs);
- }
+ pipesConfig.setJvmArgs(forkedJvmArgs(conf));
// an HtmlMapper can't be passed to the forked JVM, which always uses Tika's default
if (!DefaultHtmlMapper.class.equals(htmlMapperClass)) {
@@ -600,6 +603,26 @@ public class ParserBolt extends BaseRichBolt {
}
}
+ /**
+ * Returns the JVM arguments of a forked process: the configured ones, plus {@link
+ * #PIPES_DEFAULT_HEAP} when none of them sets a maximum heap. Without it Tika gives each fork 60% of
+ * the RAM, whatever the number of forks on the host.
+ */
+ static List<String> forkedJvmArgs(Map<String, Object> conf) {
+ final List<String> jvmArgs =
+ new ArrayList<>(ConfUtils.loadListFromConf(PIPES_JVM_ARGS_PARAM, conf));
+ if (jvmArgs.stream().noneMatch(ParserBolt::setsHeap)) {
+ jvmArgs.add(PIPES_DEFAULT_HEAP);
+ }
+ return jvmArgs;
+ }
+
+ private static boolean setsHeap(String jvmArg) {
+ return jvmArg.startsWith("-Xmx")
+ || jvmArg.startsWith("-XX:MaxHeapSize")
+ || jvmArg.startsWith("-XX:MaxRAM");
+ }
+
/**
* The parse under {@link #PARSE_TIMEOUT_PARAM} did not complete within {@link #parseTimeout}.
*/
diff --git a/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java
new file mode 100644
index 00000000..1478b200
--- /dev/null
+++ b/external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltPipesJvmArgsTest.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.stormcrawler.tika;
+
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class ParserBoltPipesJvmArgsTest {
+
+ @Test
+ void defaultHeapWhenNoArgumentIsSet() {
+ Assertions.assertEquals(List.of("-Xmx512m"), ParserBolt.forkedJvmArgs(Map.of()));
+ }
+
+ @Test
+ void defaultHeapIsAddedToOtherArguments() {
+ Assertions.assertEquals(
+ List.of("-Djava.awt.headless=true", "-Xmx512m"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-Djava.awt.headless=true")));
+ }
+
+ @Test
+ void configuredHeapIsKept() {
+ Assertions.assertEquals(
+ List.of("-Xmx1g"),
+ ParserBolt.forkedJvmArgs(Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-Xmx1g")));
+ }
+
+ @Test
+ void configuredMaxHeapSizeIsKept() {
+ Assertions.assertEquals(
+ List.of("-XX:MaxHeapSize=2g"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(ParserBolt.PIPES_JVM_ARGS_PARAM, "-XX:MaxHeapSize=2g")));
+ }
+
+ @Test
+ void configuredRamPercentageIsKept() {
+ Assertions.assertEquals(
+ List.of("-XX:MaxRAMPercentage=25"),
+ ParserBolt.forkedJvmArgs(
+ Map.of(
+ ParserBolt.PIPES_JVM_ARGS_PARAM,
+ List.of("-XX:MaxRAMPercentage=25"))));
+ }
+}