You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/08/27 12:27:57 UTC
[PR] Only follow robots.txt redirects on the same host [stormcrawler]
rzo1 opened a new pull request, #2077:
URL: https://github.com/apache/stormcrawler/pull/2077
`HttpRobotRulesParser` resolved the `Location` header of a robots.txt redirect and re-fetched it without checking scheme, host or port, so the fetch could end up on any host the redirect named. It is now followed only when the target is http or https and shares scheme, host and port with the URL it was reached from; otherwise the response is treated like any other that yields no rules.
Sites serving robots.txt through a redirect to another host, a CDN for instance, are affected: `http.robots.redirect.crosshost.allow` (default false) restores the old behaviour for http and https targets. Documented in `crawler-default.yaml` and `configuration.adoc`.
### For all changes
- [ ] Is there a issue associated with this PR? Is it referenced in the commit message? - no issue
- [ ] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve? - no issue
- [x] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [x] Is your initial contribution a single, squashed commit?
- [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`?
### For code changes
- [ ] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - `core` module tests only
- [x] Have you written or updated unit tests to verify your changes? - new `HttpRobotRulesParserRedirectTargetTest`; the existing redirect test sets the new key, since its chain crosses ports
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - no new dependencies
- [x] If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a
- [x] If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Only follow robots.txt redirects on the same host [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2077:
URL: https://github.com/apache/stormcrawler/pull/2077#discussion_r3879546577
##########
core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:
##########
@@ -149,7 +170,18 @@ public BaseRobotRules getRobotRulesSet(Protocol http, URL url) {
String redirection = response.getMetadata().getFirstValue(HttpHeaders.LOCATION);
LOG.debug("Redirected from {} to {}", redir, redirection);
if (StringUtils.isNotBlank(redirection)) {
- redir = URLUtil.resolveUrl(redir, redirection);
+ URL target = URLUtil.resolveUrl(redir, redirection);
+ if (!followRedirect(redir, target)) {
+ LOG.debug(
+ "Robots for {} redirected to {} which is not fetched "
+ + "(not on the same host as {})",
+ url,
+ target,
+ redir);
+ // handled like any other response which does not provide rules
+ break;
Review Comment:
Agreed, and decided explicitly rather than by fall-through. A refused redirect now sets `cacheRule = false`, so it lands in the error cache like a 429 or a 5xx instead of the success cache, and the hosts on the chain get nothing stored under their keys either. Default is fail-closed: no rules for that host, so nothing is crawled there until the redirect is resolved. `http.robots.redirect.refused.allow: true` restores allow-all for anyone who wants it. It is logged at warn with the target, why it was not followed and which setting changes it, so the two outcomes are distinguishable from a genuinely absent robots.txt.
##########
core/src/test/java/org/apache/stormcrawler/protocol/HttpRobotRulesParserRedirectTest.java:
##########
@@ -79,6 +79,8 @@ void setUp() {
mockServer7.start();
mockServer8.start();
conf.put("http.agent.name", "this_is_only_a_test");
+ // the redirect chains tested below point at other ports of the same host
+ conf.put("http.robots.redirect.crosshost.allow", true);
Review Comment:
Added. `HttpRobotRulesParserRedirectTest` now has `testRedirectOnTheSameAuthorityIsFollowedByDefault` and `testRedirectToAnotherAuthorityIsNotFollowedByDefault`, both on the default conf, plus `testHeadersNotSentToAnUnfollowedRedirectTarget`. The same-host http -> https case is covered at unit level in `HttpRobotRulesParserRedirectTargetTest` (upgrade, explicit `:443`, explicit same port, downgrade, other host, other port) rather than end to end — core has no keystore test setup, and a genuine upgrade compares `http:h:80` against `https:h:443`, which the wiremock harness cannot represent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Only follow robots.txt redirects on the same host [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #2077:
URL: https://github.com/apache/stormcrawler/pull/2077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Only follow robots.txt redirects on the same host [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #2077:
URL: https://github.com/apache/stormcrawler/pull/2077#discussion_r3878556400
##########
core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:
##########
@@ -149,7 +170,18 @@ public BaseRobotRules getRobotRulesSet(Protocol http, URL url) {
String redirection = response.getMetadata().getFirstValue(HttpHeaders.LOCATION);
LOG.debug("Redirected from {} to {}", redir, redirection);
if (StringUtils.isNotBlank(redirection)) {
- redir = URLUtil.resolveUrl(redir, redirection);
+ URL target = URLUtil.resolveUrl(redir, redirection);
+ if (!followRedirect(redir, target)) {
+ LOG.debug(
+ "Robots for {} redirected to {} which is not fetched "
+ + "(not on the same host as {})",
+ url,
+ target,
+ redir);
+ // handled like any other response which does not provide rules
+ break;
Review Comment:
The new tests make rejected redirect -> allow-all explicit, and that 3xx is then cached as a successful `EMPTY_RULES` result for 6h. I don't think an egress restriction should silently turn into permission to crawl without rules. Could we at least keep it out of the success cache and decide explicitly between fail-closed and allow-all?
##########
core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:
##########
@@ -44,6 +44,8 @@ public class HttpRobotRulesParser extends RobotRulesParser {
protected boolean allow5xx = false;
+ protected boolean allowCrossHostRedirects = false;
Review Comment:
Two cases hide behind `false`. `getCacheKey` treats `http:h:80` and `https:h:443` as different authorities, so the scheme upgrade on the same host dies on the first hop; no other host is involved, I'd always follow that one. Apex -> www and CDNs are a real host change: the robots RFC says a crawler should follow those (hence our five-hop cap), but it's a SHOULD, and an egress restriction is defensible under our security model. Keep it opt-in if you want, but exempt the same-host scheme upgrade?
##########
core/src/test/java/org/apache/stormcrawler/protocol/HttpRobotRulesParserRedirectTest.java:
##########
@@ -79,6 +79,8 @@ void setUp() {
mockServer7.start();
mockServer8.start();
conf.put("http.agent.name", "this_is_only_a_test");
+ // the redirect chains tested below point at other ports of the same host
+ conf.put("http.robots.redirect.crosshost.allow", true);
Review Comment:
With the flag on here nothing runs end to end with the default, and `localhost:8090 -> :8091` is exactly the chain that now ends as allow-all. One case without the flag? Same host `http -> https` in the new test would be good too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Only follow robots.txt redirects on the same host [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2077:
URL: https://github.com/apache/stormcrawler/pull/2077#discussion_r3879541808
##########
core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:
##########
@@ -44,6 +44,8 @@ public class HttpRobotRulesParser extends RobotRulesParser {
protected boolean allow5xx = false;
+ protected boolean allowCrossHostRedirects = false;
Review Comment:
Exempted, one way only. `isSchemeUpgrade` follows http -> https on the same host when the port is unchanged or both sides use their scheme default, so `http://h/robots.txt -> https://h/robots.txt` and the `:443` / explicit-same-port variants are followed by default; `http://h:8080 -> https://h:9443` stays refused. The downgrade is deliberately not exempt, since it would put the configured `http.basicauth.*` / `http.custom.headers` on the wire in clear and let rules read over plain text replace the ones the redirect was reached from. Also renamed the flag to `http.robots.redirect.crossorigin.allow` while it is still unreleased — it governs scheme and port as well as host, and one scheme change is no longer cross-anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org