You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/08/29 11:27:14 UTC
[I] Politeness queues and the robots.txt cache key on the raw host string, so percent-escaped host aliases get their own queue and their own robots.txt fetch [stormcrawler]
rzo1 opened a new issue, #2082:
URL: https://github.com/apache/stormcrawler/issues/2082
## What happens
`FetchItem.create` uses `u.getHost()` as the politeness queue id and `HttpRobotRulesParser.getCacheKey` builds `protocol:host:port` from the same string. Both only lowercase it. okhttp percent-decodes the host when it parses the URL, so `http://exampl%65.org/` and `http://example.org/` are one origin at connect time but two queue ids and two robots.txt cache entries in the bolt. `BasicURLNormalizer` does not close the gap either: it lowercases the host but leaves percent-escapes in place.
## Where
`core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java:189` and `:197` (the same lowercasing at `:170`, and `SimpleFetcherBolt.java:628` and `:631`), `core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:69-81`, `core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLNormalizer.java:136`. Config keys involved: `fetcher.queue.mode` (default `byHost`), `fetcher.server.delay`, `fetcher.threads.number`.
```java
key = u.getHost();
...
queueId = key.toLowerCase(Locale.ROOT);
```
```java
String host = url.getHost().toLowerCase(Locale.ROOT);
return protocol + ":" + host + ":" + port;
```
## Why it matters
One server can be addressed under many host spellings that all resolve to the same origin, and each spelling gets its own delay clock and its own robots.txt download. The per-host delay is then applied several times in parallel to the same server, bounded by `fetcher.threads.number` (10 by default), not by the number of aliases. Robots rules are still obeyed, since every alias re-fetches and re-parses the same file, so this is wasted requests and a politeness overrun on the remote side rather than a rules bypass. A crawl that follows outlinks can pick these up from any page that links to the aliased forms.
## Reproduction
Save as `core/src/test/java/org/apache/stormcrawler/protocol/HostAliasCacheKeyTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.protocol;
import java.net.URL;
import okhttp3.HttpUrl;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
/**
* Two URLs whose host strings differ only by percent-escaping or by a trailing dot are sent to the
* same server by okhttp, so they must share one robots.txt cache entry.
*/
class HostAliasCacheKeyTest {
@Test
void okhttpCollapsesHostAliases() {
// what the client actually connects to
Assertions.assertEquals(
"example.org", HttpUrl.parse("http://%65xample.org/a").host(), "percent-escaped");
Assertions.assertEquals(
"example.org", HttpUrl.parse("http://exampl%65.org/a").host(), "percent-escaped");
Assertions.assertEquals(
"example.org", HttpUrl.parse("http://EXAMPLE.org/a").host(), "upper case");
}
@Test
void robotsCacheKeyIsTheSameForHostAliases() throws Exception {
String canonical = HttpRobotRulesParser.getCacheKey(new URL("http://example.org/a"));
String escaped = HttpRobotRulesParser.getCacheKey(new URL("http://exampl%65.org/a"));
Assertions.assertEquals(canonical, escaped);
}
}
```
Run it:
```
mvn -pl core test -Dtest=HostAliasCacheKeyTest
```
`okhttpCollapsesHostAliases` passes and shows what the client connects to; `robotsCacheKeyIsTheSameForHostAliases` fails on main.
```
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.151 s <<< FAILURE! -- in org.apache.stormcrawler.protocol.HostAliasCacheKeyTest
[ERROR] org.apache.stormcrawler.protocol.HostAliasCacheKeyTest.robotsCacheKeyIsTheSameForHostAliases -- Time elapsed: 0.018 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <http:example.org:80> but was: <http:exampl%65.org:80>
```
## Suggested fix
Add one host canonicalisation helper in `URLUtil` that percent-decodes the host, lowercases it and strips a trailing dot, and call it from `FetchItem.create`, `SimpleFetcherBolt.getPolitenessKey` and `HttpRobotRulesParser.getCacheKey` so all three agree with what okhttp will connect to. Alternatively reject URLs whose host contains a percent escape in `BasicURLNormalizer.filter`, which is stricter and drops such URLs from the crawl entirely; that changes what an existing crawl accepts, so it needs a release note. The first option is the smaller change and does not alter which URLs are crawled, only how they are grouped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Politeness queues and the robots.txt cache key on the raw host string, so percent-escaped host aliases get their own queue and their own robots.txt fetch [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 closed issue #2082: Politeness queues and the robots.txt cache key on the raw host string, so percent-escaped host aliases get their own queue and their own robots.txt fetch
URL: https://github.com/apache/stormcrawler/issues/2082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Politeness queues and the robots.txt cache key on the raw host string, so percent-escaped host aliases get their own queue and their own robots.txt fetch [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on issue #2082:
URL: https://github.com/apache/stormcrawler/issues/2082#issuecomment-5467103729
Hi! I'd like to take this up.
Plan: I'll normalize percent-escaped hosts before keying the politeness queues and the robots.txt cache, so the same host doesn't get treated as different hosts — with regression tests.
/assign take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org