You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/09/18 19:51:04 UTC
[PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
rzo1 opened a new pull request, #2169:
URL: https://github.com/apache/stormcrawler/pull/2169
The fetcher stores response headers prefixed with `protocol.md.prefix`, so reading `Content-Type` without it never found a value in a shipped topology. The declared type was ignored and every sitemap was identified by guessing. It also left the media type check in the content sniffing dead: a page served as `text/html` carrying the sitemaps namespace was promoted to a sitemap.
Reads the key with the configured prefix, as `JSoupParserBolt` does.
The other half of #2100 is not addressed here. `processGzippedXML` streams into SAX, so there is no decompressed buffer to bound — what grows is the accumulated `SiteMapURL` set, and neither `MAX_BYTES_ALLOWED` nor `MAX_URLS` is applied on the XML path. Raised upstream as crawler-commons/crawler-commons#611, with #2166 for a cap on our side in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 merged PR #2169:
URL: https://github.com/apache/stormcrawler/pull/2169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2169:
URL: https://github.com/apache/stormcrawler/pull/2169#discussion_r4053779464
##########
core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:
##########
@@ -111,13 +112,16 @@ public class SiteMapParserBolt extends StatusEmitterBolt {
private List<Extension> extensionsToParse;
+ /** Prefix under which the fetcher stores the response headers, see protocol.md.prefix. */
+ private String protocolMetadataPrefix = "";
+
@Override
public void execute(Tuple tuple) {
Metadata metadata = (Metadata) tuple.getValueByField("metadata");
byte[] content = tuple.getBinaryByField("content");
String url = tuple.getStringByField("url");
- String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE);
+ String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE, protocolMetadataPrefix);
Review Comment:
Good catch. crawler-commons gets null again and detects the format itself, the header is only used for the sniff check. Added a test with a gzipped sitemap declared as `application/xml` with `Content-Encoding: x-gzip`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #2169:
URL: https://github.com/apache/stormcrawler/pull/2169#discussion_r4052757741
##########
core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:
##########
@@ -111,13 +112,16 @@ public class SiteMapParserBolt extends StatusEmitterBolt {
private List<Extension> extensionsToParse;
+ /** Prefix under which the fetcher stores the response headers, see protocol.md.prefix. */
+ private String protocolMetadataPrefix = "";
+
@Override
public void execute(Tuple tuple) {
Metadata metadata = (Metadata) tuple.getValueByField("metadata");
byte[] content = tuple.getBinaryByField("content");
String url = tuple.getStringByField("url");
- String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE);
+ String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE, protocolMetadataPrefix);
Review Comment:
the type now also reaches crawler-commons, which trusts a declared `application/xml` instead of detecting it: a gzipped sitemap served as xml with `Content-Encoding: x-gzip` (okhttp only decodes `gzip`) parsed before and fails now. keep passing null here and use the header for the sniff check only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org