You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/08/29 11:27:10 UTC
[I] Nothing in the shipped configuration keeps the fetcher off private address space [stormcrawler]
rzo1 opened a new issue, #2080:
URL: https://github.com/apache/stormcrawler/issues/2080
## What happens
The connection-time IP filter is the only check that looks at the address the fetcher actually connects to, and both of its keys ship commented out in `crawler-default.yaml`. `HttpProtocol` installs the interceptor only when `IPFilterRules.isEmpty()` is false, so at defaults it is never installed. What remains is the regex exclusion list in the archetypes, which matches literal `localhost`, dotted-quad 127/8, 10/8, 192.168/16, 172.16/12 and `[::1]` and nothing else. Link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 host forms that the JVM resolver still maps to 127.0.0.1 all pass it.
## Where
`core/src/main/resources/crawler-default.yaml:161-162`, under the comment block at `:150-160`, keys `http.filter.ipaddress.include` and `http.filter.ipaddress.exclude`:
```yaml
# http.filter.ipaddress.include:
# http.filter.ipaddress.exclude: "localhost,sitelocal"
```
`core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java:234-237`:
```java
final IPFilterRules ipFilterRules = new IPFilterRules(conf);
if (!ipFilterRules.isEmpty()) {
builder.addNetworkInterceptor(new HTTPFilterIPAddressInterceptor(ipFilterRules));
}
```
The regex rules are at `archetype/src/main/resources/archetype-resources/src/main/resources/default-regex-filters.txt:21-29`, byte-identical in the opensearch and solr archetypes. The comment above them states the intent: to stop faked links leaking information from services running on the crawling machine.
## Why it matters
A crawled page decides which hosts the fetcher connects to, and the fetched body goes on to be parsed and indexed. A string-level rule cannot enforce this at all, since it never sees where a host name resolves to, and the list of literal forms it does cover has gaps that need no DNS control. On a cloud worker the link-local range is reachable and unauthenticated. The regex list is also the only defence in the archetypes, and `urlfilters.config.file` itself ships commented out in `crawler-default.yaml:243`, so a topology built from the library rather than an archetype has no URL filtering at all.
## Reproduction
Save as `core/src/test/java/org/apache/stormcrawler/filtering/DefaultRegexFiltersPrivateRangeTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.filtering;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.filtering.regex.RegexURLFilter;
import org.apache.stormcrawler.util.URLUtil;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
/**
* The private-range rules shipped in the archetype default-regex-filters.txt, applied to hosts the
* JVM resolver maps into loopback, link-local and other non-routable space.
*/
class DefaultRegexFiltersPrivateRangeTest {
/** The exclusion rules of archetype default-regex-filters.txt, verbatim. */
private static final String[] ARCHETYPE_RULES = {
"-^(file|ftp|mailto):",
"-^https?://(?:localhost|127(?:\\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3}|\\[::1\\])(?::\\d+)?(?:/|$)",
"-^https?://(?:10(?:\\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3})(?::\\d+)?(?:/|$)",
"-^https?://(?:192\\.168(?:\\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\\d+)?(?:/|$)",
"-^https?://(?:172\\.(?:1[6789]|2[0-9]|3[01])(?:\\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\\d+)?(?:/|$)",
"+."
};
private URLFilter createFilter() {
ObjectNode filterParams = new ObjectNode(JsonNodeFactory.instance);
ArrayNode rules = filterParams.putArray("urlFilters");
for (String rule : ARCHETYPE_RULES) {
rules.add(rule);
}
RegexURLFilter filter = new RegexURLFilter();
Map<String, Object> conf = new HashMap<>();
filter.configure(conf, filterParams);
return filter;
}
private void assertRejected(URLFilter filter, String url) throws MalformedURLException {
URL source = URLUtil.toURL("http://www.example.com/index.html");
Assertions.assertNull(filter.filter(source, new Metadata(), url), url);
}
@Test
void dottedQuadFormsAreRejected() throws MalformedURLException {
URLFilter filter = createFilter();
assertRejected(filter, "http://127.0.0.1/");
assertRejected(filter, "http://10.0.0.5/");
assertRejected(filter, "http://192.168.1.1/");
}
@Test
void otherNonRoutableRangesAreRejected() throws MalformedURLException {
URLFilter filter = createFilter();
assertRejected(filter, "http://169.254.169.254/");
assertRejected(filter, "http://100.64.0.1/");
assertRejected(filter, "http://[fd00::1]/");
}
/** Both host forms are resolved to 127.0.0.1 by InetAddress.getByName. */
@Test
void abbreviatedAndIntegerLoopbackFormsAreRejected() throws MalformedURLException {
URLFilter filter = createFilter();
assertRejected(filter, "http://127.1/");
assertRejected(filter, "http://2130706433/");
}
}
```
Run it:
```
mvn -pl core test -Dtest=DefaultRegexFiltersPrivateRangeTest
```
The archetype exclusion rules are used verbatim and two of the three tests fail on main. `InetAddress.getByName` maps both `127.1` and `2130706433` to 127.0.0.1 on the JDK used here (Temurin 25).
```
[ERROR] Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
[ERROR] DefaultRegexFiltersPrivateRangeTest.otherNonRoutableRangesAreRejected:77
http://169.254.169.254/ ==> expected: <null> but was: <http://169.254.169.254/>
[ERROR] DefaultRegexFiltersPrivateRangeTest.abbreviatedAndIntegerLoopbackFormsAreRejected:86
http://127.1/ ==> expected: <null> but was: <http://127.1/>
```
The first test, covering the dotted-quad forms the rules were written for, passes.
## Suggested fix
Ship `http.filter.ipaddress.exclude` enabled in `crawler-default.yaml` and in the three archetype `crawler-conf.yaml` files, covering loopback, RFC1918, 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8 and the IPv6 equivalents. Add the missing ranges and the abbreviated and integer IPv4 forms to the archetype `default-regex-filters.txt`, and say in the comment there that a regex list cannot stop a host name that resolves into private space. Since the two layers judge different bytes, also make `BasicURLNormalizer` canonicalise the authority: `BasicURLNormalizer.java:135-150` lower-cases the host and converts an IDN host to ASCII, but does nothing else, so a percent-encoded or abbreviated host reaches the filters in a different form from the one the resolver sees. Enabling the exclude list by default changes behaviour for anyone crawling an intranet or a loopback service, so it needs a release note and a documented way to opt out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Nothing in the shipped configuration keeps the fetcher off private address space [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on issue #2080:
URL: https://github.com/apache/stormcrawler/issues/2080#issuecomment-5467102734
Hi! I'd like to take this issue.
Plan: I'll ship a default configuration that blocks the fetcher from private address space (loopback/RFC1918/link-local) via the IP filter, and add tests verifying the default blocks these ranges.
/assign take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Nothing in the shipped configuration keeps the fetcher off private address space [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 closed issue #2080: Nothing in the shipped configuration keeps the fetcher off private address space
URL: https://github.com/apache/stormcrawler/issues/2080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org