You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by rz...@apache.org on 2026/09/19 16:35:39 UTC
(stormcrawler) branch main updated: #2095 Apply the IP address filter to the requests made by the Playwright browser (#2165)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
The following commit(s) were added to refs/heads/main by this push:
new 78ea4a8f #2095 Apply the IP address filter to the requests made by the Playwright browser (#2165)
78ea4a8f is described below
commit 78ea4a8f732c03f4bf7c8eeeb8bec9995e20b07a
Author: Richard Zowalla <13...@users.noreply.github.com>
AuthorDate: Sat Sep 19 18:35:34 2026 +0200
#2095 Apply the IP address filter to the requests made by the Playwright browser (#2165)
* #2095 Apply the IP address filter to the requests made by the Playwright browser.
* Parse browser URLs leniently, cover popups and block service workers in the Playwright IP filter.
---
core/src/main/resources/crawler-default.yaml | 5 +-
docs/src/main/asciidoc/configuration.adoc | 16 +++-
.../protocol/playwright/HttpProtocol.java | 82 ++++++++++++++++++-
.../protocol/playwright/IPFilterTest.java | 93 ++++++++++++++++++++++
4 files changed, 190 insertions(+), 6 deletions(-)
diff --git a/core/src/main/resources/crawler-default.yaml b/core/src/main/resources/crawler-default.yaml
index 78478737..e171f35c 100644
--- a/core/src/main/resources/crawler-default.yaml
+++ b/core/src/main/resources/crawler-default.yaml
@@ -245,7 +245,7 @@ config:
# like it does when redirects are not followed at all.
http.allow.redirects.max: 5
- # IP address filtering (okhttp protocol only). Optionally limit or block
+ # IP address filtering (okhttp and playwright protocols). Optionally limit or block
# connections to IP address ranges once the host name has been resolved. This
# prevents information leakage to a public index when a DNS entry points to a
# private or loopback address. Rules can be given as a comma-separated string
@@ -271,6 +271,9 @@ config:
# Fetches through a proxy are not filtered: the proxy resolves and connects
# to the target, so only its address is known here and a proxy on a private
# address is fine. The proxy's own egress rules decide what it may reach.
+ # Playwright checks the page, its subresources and popups against an address
+ # resolved separately from the browser and blocks service workers; redirect
+ # hops and WebSockets are not checked.
http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128"
# Allow all if robots.txt cannot be parsed due to code 403 (Forbidden):
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index 157a9b21..19df7288 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -264,8 +264,8 @@ header.
| http.accept.language | en-us,en-gb,en;q=0.7,*;q=0.3 | HTTP Accept-Language
header.
| http.content.partial.as.trimmed | false | Accepts partially fetched content in OKHTTP.
-| http.filter.ipaddress.include | - | (OkHttp only) Comma-separated list (or YAML list) of allowed IP ranges. If empty, all addresses are allowed unless excluded. See <<IP Address Filtering>>.
-| http.filter.ipaddress.exclude | localhost, sitelocal, linklocal, anylocal, multicast, 100.64.0.0/10, 0.0.0.0/8, fc00::/7, ::/128 | (OkHttp only) Comma-separated list (or YAML list) of blocked IP ranges. Set to `""` to crawl private address space. See <<IP Address Filtering>>.
+| http.filter.ipaddress.include | - | (OkHttp and Playwright) Comma-separated list (or YAML list) of allowed IP ranges. If empty, all addresses are allowed unless excluded. See <<IP Address Filtering>>.
+| http.filter.ipaddress.exclude | localhost, sitelocal, linklocal, anylocal, multicast, 100.64.0.0/10, 0.0.0.0/8, fc00::/7, ::/128 | (OkHttp and Playwright) Comma-separated list (or YAML list) of blocked IP ranges. Set to `""` to crawl private address space. See <<IP Address Filtering>>.
| http.trust.everything | false | (OkHttp only) If true, accept any TLS certificate, including self-signed, expired or otherwise invalid ones. The servers are then not authenticated: anyone able to answer for the host name receives everything sent to them. Credentials (basic auth, credential headers, cookies) are withheld from cleartext http:// requests and from https:// requests on such connections unless `http.credentials.allow.insecure` is set to true.
| http.verify.hostnames | true | (OkHttp only) If true, check that the certificate presented by the server matches the host name contacted. Independent of `http.trust.everything`: a valid certificate for a different name is accepted when this is false. Credentials are then withheld unless `http.credentials.allow.insecure` is set to true.
| http.credentials.allow.insecure | false | (OkHttp only) If true, send credentials (basic auth, credential headers, cookies) even on connections which do not authenticate the server: cleartext http:// urls and, when `http.trust.everything` is true or `http.verify.hostnames` is false, https:// urls as well. Redirect hops to another origin (scheme, host or port) never carry them.
@@ -290,6 +290,13 @@ Fetches through a proxy are not filtered. The proxy resolves the target host and
the protocol only sees the address of the proxy, which may well be private. What the fetcher can
reach through a proxy is then decided by the egress rules of the proxy.
+The Playwright protocol (`org.apache.stormcrawler.protocol.playwright.HttpProtocol`) applies the same
+rules to the requests of the page, its subresources and popups, and aborts the rejected ones.
+Service workers are blocked while the filter is active, as their requests cannot be intercepted.
+Redirect hops and WebSocket connections are not checked. The host is resolved separately from the
+browser, so short-lived DNS records or a remote browser using another resolver can lead to a
+different address.
+
Two properties control the behaviour:
* `http.filter.ipaddress.include` — defines the allowed IP ranges. If empty, all addresses are
@@ -318,8 +325,9 @@ http.filter.ipaddress.include: "10.0.0.0/8"
grade NAT (`100.64.0.0/10`) and IPv6 unique local addresses (`fd00::/8`) have no keyword and are added
as CIDR blocks when they are also to be blocked.
-When a connection to a blocked address is attempted, the fetch fails with an `IOException` and a
-warning is logged. If neither property is set, no IP filtering is performed.
+When a connection to a blocked address is attempted, a warning is logged. With OkHttp the fetch
+fails with an `IOException`. With Playwright the request is aborted: a blocked page fails the fetch,
+a blocked subresource is missing from the rendered page. If neither property is set, no IP filtering is performed.
NOTE: When a proxy is configured, the connection is established to the proxy and the filter sees the
proxy's IP address rather than the target host's resolved address. IP filtering is therefore
diff --git a/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java b/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java
index c24bab16..929d5c89 100644
--- a/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java
+++ b/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java
@@ -29,8 +29,11 @@ import com.microsoft.playwright.Request;
import com.microsoft.playwright.Tracing;
import com.microsoft.playwright.options.HttpHeader;
import com.microsoft.playwright.options.Proxy;
+import com.microsoft.playwright.options.ServiceWorkerPolicy;
import com.microsoft.playwright.options.WaitUntilState;
+import java.net.InetAddress;
import java.net.URI;
+import java.net.UnknownHostException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
@@ -40,12 +43,14 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import okhttp3.HttpUrl;
import org.apache.commons.lang3.StringUtils;
import org.apache.storm.Config;
import org.apache.storm.utils.MutableInt;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.persistence.Status;
import org.apache.stormcrawler.protocol.AbstractHttpProtocol;
+import org.apache.stormcrawler.protocol.IPFilterRules;
import org.apache.stormcrawler.protocol.Protocol;
import org.apache.stormcrawler.protocol.ProtocolResponse;
import org.apache.stormcrawler.util.ConfUtils;
@@ -84,6 +89,8 @@ public class HttpProtocol extends AbstractHttpProtocol {
private PageActions pageActions = PageActions.emptyPageActions;
+ private IPFilterRules ipFilterRules;
+
@Override
public void configure(final Config conf) {
super.configure(conf);
@@ -145,12 +152,30 @@ public class HttpProtocol extends AbstractHttpProtocol {
overrideStatusOnContent =
ConfUtils.getBoolean(conf, "playwright.override.status.on.content", false);
+ configureIPFilter(conf);
+
final NewContextOptions b_c_options = buildContextOptions(conf, getAgentString(conf));
context = browser.newContext(b_c_options);
context.setDefaultTimeout(timeout);
+ // on the context so that popups are covered too, the page route falls back to it
+ if (ipFilterRules != null) {
+ context.route(
+ lambdaUrl -> true,
+ route -> {
+ if (isAllowedAddress(route.request().url())) {
+ route.resume();
+ } else {
+ LOG.warn(
+ "Blocked request to forbidden IP address: {}",
+ route.request().url());
+ route.abort("addressunreachable");
+ }
+ });
+ }
+
// list of resource types to skip
// document, stylesheet, image, media, font, script, texttrack, xhr, fetch,
// eventsource, websocket, manifest, other
@@ -168,6 +193,55 @@ public class HttpProtocol extends AbstractHttpProtocol {
pageActions = PageActions.fromConf(conf);
}
+ /**
+ * Sets up the filtering of the requests made by the browser against http.filter.ipaddress.*. As
+ * with the OkHttp protocol, fetches through a proxy are not filtered.
+ */
+ void configureIPFilter(final Config conf) {
+ final IPFilterRules rules = new IPFilterRules(conf);
+ if (rules.isEmpty()) {
+ ipFilterRules = null;
+ } else if (StringUtils.isNotBlank(ConfUtils.getString(conf, "http.proxy"))) {
+ ipFilterRules = null;
+ LOG.info(
+ "http.filter.ipaddress.* do not apply to fetches through a proxy, the proxy"
+ + " resolves the target host and its own egress rules decide which"
+ + " addresses are reached");
+ } else {
+ ipFilterRules = rules;
+ }
+ }
+
+ /**
+ * Checks the host of a request made by the browser against the IP filter rules. The host is
+ * resolved here, separately from the browser, so the address the browser connects to can
+ * differ, e.g. with short-lived DNS records or a remote browser using another resolver.
+ *
+ * @return false if any address of the host is rejected or the host cannot be resolved
+ */
+ boolean isAllowedAddress(final String url) {
+ if (ipFilterRules == null) {
+ return true;
+ }
+ // lenient like the browser, e.g. with '|' in the path or '_' in the host name
+ final HttpUrl parsed = HttpUrl.parse(url);
+ if (parsed == null) {
+ // data:, blob: and the like do not open a connection
+ return !StringUtils.startsWithIgnoreCase(url, "http:")
+ && !StringUtils.startsWithIgnoreCase(url, "https:");
+ }
+ try {
+ for (InetAddress address : InetAddress.getAllByName(parsed.host())) {
+ if (!ipFilterRules.accept(address)) {
+ return false;
+ }
+ }
+ return true;
+ } catch (UnknownHostException e) {
+ return false;
+ }
+ }
+
/**
* Builds the options of the browser context shared by all fetches, whether the browser is
* launched locally or reached via CDP or a remote Playwright server.
@@ -208,6 +282,11 @@ public class HttpProtocol extends AbstractHttpProtocol {
}
options.setIgnoreHTTPSErrors(ignoreHTTPSErrors);
+ // requests of a service worker are not routed, so they would bypass the IP filter
+ if (ipFilterRules != null) {
+ options.setServiceWorkers(ServiceWorkerPolicy.BLOCK);
+ }
+
return options;
}
@@ -273,7 +352,8 @@ public class HttpProtocol extends AbstractHttpProtocol {
route.request().resourceType())) {
route.abort();
} else {
- route.resume();
+ // lets the IP filter on the context check the request
+ route.fallback();
}
});
diff --git a/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/IPFilterTest.java b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/IPFilterTest.java
new file mode 100644
index 00000000..e6b6a633
--- /dev/null
+++ b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/IPFilterTest.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.protocol.playwright;
+
+import com.microsoft.playwright.options.ServiceWorkerPolicy;
+import org.apache.storm.Config;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/** Checks the IP filter applied to the requests of the browser. Does not need a browser. */
+class IPFilterTest {
+
+ private HttpProtocol protocol(final Config conf) {
+ final HttpProtocol protocol = new HttpProtocol();
+ protocol.configureIPFilter(conf);
+ return protocol;
+ }
+
+ private Config excludeLocal() {
+ final Config conf = new Config();
+ conf.put("http.filter.ipaddress.exclude", "localhost,sitelocal");
+ return conf;
+ }
+
+ @Test
+ void everythingAllowedWithoutRules() {
+ final HttpProtocol protocol = protocol(new Config());
+ Assertions.assertTrue(protocol.isAllowedAddress("http://127.0.0.1/"));
+ }
+
+ @Test
+ void excludedAddressesAreRejected() {
+ final HttpProtocol protocol = protocol(excludeLocal());
+ Assertions.assertFalse(protocol.isAllowedAddress("http://127.0.0.1:8080/page.html"));
+ Assertions.assertFalse(protocol.isAllowedAddress("https://localhost/"));
+ Assertions.assertFalse(protocol.isAllowedAddress("http://[::1]/"));
+ Assertions.assertFalse(protocol.isAllowedAddress("http://192.168.1.1/"));
+ Assertions.assertTrue(protocol.isAllowedAddress("http://8.8.8.8/"));
+ }
+
+ @Test
+ void urlsWithoutConnectionAreAllowed() {
+ final HttpProtocol protocol = protocol(excludeLocal());
+ Assertions.assertTrue(protocol.isAllowedAddress("data:text/plain,hello"));
+ }
+
+ @Test
+ void urlsChromiumSendsAsIsAreParsed() {
+ final HttpProtocol protocol = protocol(excludeLocal());
+ Assertions.assertFalse(protocol.isAllowedAddress("http://127.0.0.1/a|b[c]?q={x}^"));
+ Assertions.assertTrue(protocol.isAllowedAddress("http://8.8.8.8/a|b[c]?q={x}^"));
+ }
+
+ @Test
+ void serviceWorkersBlockedOnlyWithRules() {
+ Assertions.assertEquals(
+ ServiceWorkerPolicy.BLOCK,
+ protocol(excludeLocal())
+ .buildContextOptions(excludeLocal(), "test")
+ .serviceWorkers);
+ Assertions.assertNull(
+ protocol(new Config()).buildContextOptions(new Config(), "test").serviceWorkers);
+ }
+
+ @Test
+ void unresolvableHostsAreRejected() {
+ final HttpProtocol protocol = protocol(excludeLocal());
+ Assertions.assertFalse(protocol.isAllowedAddress("http://does-not-exist.invalid/"));
+ }
+
+ @Test
+ void notAppliedThroughProxy() {
+ final Config conf = excludeLocal();
+ conf.put("http.proxy", "http://proxy.example.com:3128");
+ final HttpProtocol protocol = protocol(conf);
+ Assertions.assertTrue(protocol.isAllowedAddress("http://127.0.0.1/"));
+ }
+}