You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/06/14 17:42:12 UTC
[PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
rzo1 opened a new pull request, #1943:
URL: https://github.com/apache/stormcrawler/pull/1943
text/plain content requires no markup parsing, so instead of raising an error (the default with jsoup.treat.non.html.as.error) the bolt now uses the decoded content directly as the extracted text and emits no outlinks.
Adds a unit test and documentation updates.
Closes #466
### For all changes
- [x] Is there a issue associated with this PR? Is it referenced in the commit message?
- [x] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve?
- [x] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [x] Is your initial contribution a single, squashed commit?
- [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`?
### For code changes
- [x] Have you ensured that the full suite of tests is executed via `mvn clean verify`?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file?
### Note
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#discussion_r3418782414
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -272,79 +280,91 @@ public void execute(Tuple tuple) {
try {
String html = Charset.forName(charset).decode(ByteBuffer.wrap(content)).toString();
- jsoupDoc = Parser.htmlParser().parseInput(html, url);
-
- if (!robotsMetaSkip) {
- // extracts the robots directives from the meta tags
- Element robotelement = jsoupDoc.selectFirst("meta[name~=(?i)robots][content]");
- if (robotelement != null) {
- robotsTags.extractMetaTags(robotelement.attr("content"));
- }
- }
-
- // store a normalised representation in metadata
- // so that the indexer is aware of it
- robotsTags.normaliseToMetadata(metadata);
-
- // do not extract the links if no follow has been set
- // and we are in strict mode
- if (robotsTags.isNoFollow() && robotsNoFollowStrict) {
+ if (isPlainText) {
+ // no markup to parse: the decoded content is the text itself and
+ // there are no outlinks. An empty shell document is kept so that
+ // the downstream redirection check and parse filters still work.
+ jsoupDoc = org.jsoup.nodes.Document.createShell(url);
slinks = new HashMap<>(0);
+ robotsTags.normaliseToMetadata(metadata);
+ text = html;
Review Comment:
I'd go with Option A. A plain-text file is already its own text, which is the whole point of #466, and the test asserts it's stored verbatim, newlines and all. Option B runs the content through `appendNormalisedText`, which collapses whitespace, so logs, source files and tabular dumps lose their layout. It would also force relaxing that test. For the one format where whitespace is the content, that's a poor trade.
The downside of Option A, re-reading the two config keys, is smaller than it looks. For plain text only `no.text` and `skip.after` have any effect; `include.pattern` and `exclude.tags` need markup, so there's nothing else to honor. If the custom `textextractor.class` case still bothers you, a cleaner option is a `text(String)` overload on the extractor that applies those two limits without normalizing, so both code paths share one implementation. I'd treat that as a follow-up, not part of this PR.
One open question: should plain text use the `TextExtractor` knobs at all, or is `http.content.limit` the more honest bound for raw bytes? Either way, Option A plus a test for the `skip.after` truncation works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#discussion_r3415209594
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -272,79 +280,91 @@ public void execute(Tuple tuple) {
try {
String html = Charset.forName(charset).decode(ByteBuffer.wrap(content)).toString();
- jsoupDoc = Parser.htmlParser().parseInput(html, url);
-
- if (!robotsMetaSkip) {
- // extracts the robots directives from the meta tags
- Element robotelement = jsoupDoc.selectFirst("meta[name~=(?i)robots][content]");
- if (robotelement != null) {
- robotsTags.extractMetaTags(robotelement.attr("content"));
- }
- }
-
- // store a normalised representation in metadata
- // so that the indexer is aware of it
- robotsTags.normaliseToMetadata(metadata);
-
- // do not extract the links if no follow has been set
- // and we are in strict mode
- if (robotsTags.isNoFollow() && robotsNoFollowStrict) {
+ if (isPlainText) {
+ // no markup to parse: the decoded content is the text itself and
+ // there are no outlinks. An empty shell document is kept so that
+ // the downstream redirection check and parse filters still work.
+ jsoupDoc = org.jsoup.nodes.Document.createShell(url);
slinks = new HashMap<>(0);
+ robotsTags.normaliseToMetadata(metadata);
+ text = html;
Review Comment:
the plain text path stores `html` directly, so `textextractor.skip.after` and `textextractor.no.text` don't apply here. fine for typical .txt, but with
`http.content.limit: -1` (default) a large file gets stored in full. worth truncating to maxTextSize, or noting the limitation in the docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#discussion_r3419547766
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -272,79 +280,91 @@ public void execute(Tuple tuple) {
try {
String html = Charset.forName(charset).decode(ByteBuffer.wrap(content)).toString();
- jsoupDoc = Parser.htmlParser().parseInput(html, url);
-
- if (!robotsMetaSkip) {
- // extracts the robots directives from the meta tags
- Element robotelement = jsoupDoc.selectFirst("meta[name~=(?i)robots][content]");
- if (robotelement != null) {
- robotsTags.extractMetaTags(robotelement.attr("content"));
- }
- }
-
- // store a normalised representation in metadata
- // so that the indexer is aware of it
- robotsTags.normaliseToMetadata(metadata);
-
- // do not extract the links if no follow has been set
- // and we are in strict mode
- if (robotsTags.isNoFollow() && robotsNoFollowStrict) {
+ if (isPlainText) {
+ // no markup to parse: the decoded content is the text itself and
+ // there are no outlinks. An empty shell document is kept so that
+ // the downstream redirection check and parse filters still work.
+ jsoupDoc = org.jsoup.nodes.Document.createShell(url);
slinks = new HashMap<>(0);
+ robotsTags.normaliseToMetadata(metadata);
+ text = html;
Review Comment:
I went for Option A now. Happy for re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche merged PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#issuecomment-4767598536
thanks @rzo1 and @dpol1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#discussion_r3415209594
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -272,79 +280,91 @@ public void execute(Tuple tuple) {
try {
String html = Charset.forName(charset).decode(ByteBuffer.wrap(content)).toString();
- jsoupDoc = Parser.htmlParser().parseInput(html, url);
-
- if (!robotsMetaSkip) {
- // extracts the robots directives from the meta tags
- Element robotelement = jsoupDoc.selectFirst("meta[name~=(?i)robots][content]");
- if (robotelement != null) {
- robotsTags.extractMetaTags(robotelement.attr("content"));
- }
- }
-
- // store a normalised representation in metadata
- // so that the indexer is aware of it
- robotsTags.normaliseToMetadata(metadata);
-
- // do not extract the links if no follow has been set
- // and we are in strict mode
- if (robotsTags.isNoFollow() && robotsNoFollowStrict) {
+ if (isPlainText) {
+ // no markup to parse: the decoded content is the text itself and
+ // there are no outlinks. An empty shell document is kept so that
+ // the downstream redirection check and parse filters still work.
+ jsoupDoc = org.jsoup.nodes.Document.createShell(url);
slinks = new HashMap<>(0);
+ robotsTags.normaliseToMetadata(metadata);
+ text = html;
Review Comment:
the plain text path stores `html` directly, so `textextractor.skip.after` and `textextractor.no.text` don't apply here. fine for typical .txt, but with `http.content.limit: -1` (default) a large file gets stored in full. worth truncating to maxTextSize, or noting the limitation in the docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1943:
URL: https://github.com/apache/stormcrawler/pull/1943#discussion_r3415389865
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -272,79 +280,91 @@ public void execute(Tuple tuple) {
try {
String html = Charset.forName(charset).decode(ByteBuffer.wrap(content)).toString();
- jsoupDoc = Parser.htmlParser().parseInput(html, url);
-
- if (!robotsMetaSkip) {
- // extracts the robots directives from the meta tags
- Element robotelement = jsoupDoc.selectFirst("meta[name~=(?i)robots][content]");
- if (robotelement != null) {
- robotsTags.extractMetaTags(robotelement.attr("content"));
- }
- }
-
- // store a normalised representation in metadata
- // so that the indexer is aware of it
- robotsTags.normaliseToMetadata(metadata);
-
- // do not extract the links if no follow has been set
- // and we are in strict mode
- if (robotsTags.isNoFollow() && robotsNoFollowStrict) {
+ if (isPlainText) {
+ // no markup to parse: the decoded content is the text itself and
+ // there are no outlinks. An empty shell document is kept so that
+ // the downstream redirection check and parse filters still work.
+ jsoupDoc = org.jsoup.nodes.Document.createShell(url);
slinks = new HashMap<>(0);
+ robotsTags.normaliseToMetadata(metadata);
+ text = html;
Review Comment:
Good catch. The plain-text branch does `text = html` directly, so neither `textextractor.no.text` nor `textextractor.skip.after` applies, and with the default `http.content.limit: -1` a large `.txt` is stored in full. I see two ways to bound it. Both reuse the existing `TextExtractor` config keys so there is no new knob to document.
### Option A: truncate the raw decoded string
Read the two limits in `prepare()` and apply them in the branch:
```java
// fields
private boolean plainTextNoText;
private int plainTextMaxSize;
// prepare()
plainTextNoText = ConfUtils.getBoolean(conf, TextExtractor.NO_TEXT_PARAM_NAME, false);
plainTextMaxSize = ConfUtils.getInt(conf, TextExtractor.TEXT_MAX_TEXT_PARAM_NAME, -1);
// in the isPlainText branch
if (plainTextNoText) {
text = "";
} else if (plainTextMaxSize > 0 && html.length() > plainTextMaxSize) {
text = html.substring(0, plainTextMaxSize);
} else {
text = html;
}
```
**Pros:** preserves the original plain-text formatting (newlines, spacing, indentation), which is usually the point of a `.txt`; trivially cheap.
**Cons:** re-reads config keys that conceptually belong to the `TextExtractor`, so a custom `textextractor.class` with different semantics would not be consulted; the cap is on character count, not on the extractor's accumulation logic.
### Option B: route the text through the configured extractor
Wrap the decoded content in the shell document and let the extractor produce the text:
```java
jsoupDoc = Document.createShell(url);
jsoupDoc.body().appendChild(new TextNode(html));
slinks = new HashMap<>(0);
robotsTags.normaliseToMetadata(metadata);
text = textExtractor.text(jsoupDoc.body());
```
**Pros:** a single text-extraction path for HTML and plain text; automatically honors `no.text`, `skip.after`, exclude tags, and any custom `textextractor.class`.
**Cons:** the extractor normalizes whitespace (`appendNormalisedText`), collapsing newlines and runs of spaces, so the stored plain text loses its original layout. Fine for indexing, lossy if the raw structure matters.
### Trade-off in one line
A keeps plain text faithful but special-cases the limits; B unifies the code path but mangles whitespace. Either way I would add a test asserting a `>skip.after` input is truncated.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org