You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "abhinav-phi (via GitHub)" <gi...@apache.org> on 2026/09/05 23:29:42 UTC
[PR] Keep the fetcher off private address space by default [stormcrawler]
abhinav-phi opened a new pull request, #2127:
URL: https://github.com/apache/stormcrawler/pull/2127
Fixes #2080.
The connection-time IP filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out in `crawler-default.yaml`, so at defaults the interceptor was never installed. The regex exclusion list in the archetypes matched literal `localhost` and four dotted-quad ranges and nothing else — link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the JVM resolver maps to 127.0.0.1 all passed it, and `InetAddress.getByName` maps `127.1` and `2130706433` to loopback without any DNS control.
- `crawler-default.yaml` ships `http.filter.ipaddress.exclude` enabled: `localhost,sitelocal,linklocal,100.64.0.0/10,fd00::/8`. A fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services (e.g. cloud instance metadata at 169.254.169.254, reachable whatever DNS name a link used, since the check runs on the resolved socket address) which a public index must not leak into
- crawling an intranet or a loopback service needs an explicit opt-out — the comment next to the key documents how
- the archetype `default-regex-filters.txt` (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 `fc00::/7` and `fe80::/10`, abbreviated loopback (`127.1`) and integer IPv4 hosts (`2130706433`), and states that a regex list judges the bytes of the URL only — it can never enforce where a host name resolves, so the IP filter stays the authoritative check
**Release note needed:** enabling the exclude list by default changes behaviour for anyone crawling an intranet or a loopback service; the way to opt out is documented next to the key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-5574853127
Red on the `rat` job — `CustomImportOrder`, 12 violations (unsorted imports, extra blank line in the import group). Since `build` is declared `needs: rat`, the test matrix never runs, so this PR still has no real CI signal and I can't review it on substance yet.
Before pushing, per `README.md` ("Note for developers"):
```
mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false
```
Same issue on eight other open PRs of yours — longer explanation in https://github.com/apache/stormcrawler/pull/2119#issuecomment-5574839248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "abhinav-phi (via GitHub) via dev" <de...@stormcrawler.apache.org>.
abhinav-phi commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-6627755370
Both landed in the merged `default-regex-filters.txt`: the IPv6 unique-local/link-local rule carries `(?i)`, and there is a dedicated `[::ffff:127.` / `[0:0:0:0:0:0:0:1]` loopback rule, so the mapped and expanded spellings no longer slip through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#discussion_r3976680622
##########
core/src/main/resources/crawler-default.yaml:
##########
@@ -165,8 +165,14 @@ config:
# topology from starting.
# Only addresses matching an include rule are fetched (empty means all are
# allowed), addresses matching an exclude rule are always blocked.
- # http.filter.ipaddress.include:
- # http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,fc00::/7"
+ # The exclude list is enabled by default: a fetched page decides which hosts
+ # the fetcher connects to, and loopback, private and link-local ranges host
+ # unauthenticated services (e.g. cloud instance metadata) which a public
+ # index must not leak into. To crawl an intranet, override the exclude list
+ # explicitly (exclude rules win over include rules, so setting only
+ # http.filter.ipaddress.include does not undo this default), e.g.
+ # http.filter.ipaddress.exclude: ""
+ http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128"
Review Comment:
with a proxy configured (`http.proxy.host`) the filter sees the proxy's socket address, not the target's, so an HTTP proxy on a private range now fails every fetch with this default. either skip the check when `connection.route().proxy()` is set and document that proxied fetches are unfiltered, or state that the list must be cleared behind a proxy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-5601942356
Thanks — pushed 19106ee8 closing all three items:
**Shipped policy gaps.** `http.filter.ipaddress.exclude` now includes `0.0.0.0/8` and `::/128` (value: `localhost,sitelocal,linklocal,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128`). `IPFilterRules.accept` returned true for `0.0.0.0`, `0.1.2.3` and `::` before — the regex blocks the literals but a hostname resolving into those ranges still reached them. I used `::/128` (the exact unspecified address) rather than a broad IPv6 range so no routable address is excluded. `CIDR` handles both: `contains()` short-circuits on the v4/v6 byte-length mismatch, and the mapped-address mask shift does not misfire on a `::/128`.
**Fixture restored.** `core/src/test/resources/default-regex-filters.txt` is back to its upstream content (the earlier revert had been undone by a later checkout on this branch). This PR's archetype copy is the separate `default-regex-filters-archetype.txt`, loaded by the new test, so `RegexFilterTest`/`URLFilterBoltTest` keep their original fixture.
**Test coverage.** The shipped-default test now pins every excluded range with its boundary addresses and public neighbours (9.255.255.255 / 11.0.0.0, 172.15/172.32, 192.167/192.169, 100.63/100.128, plus a public IPv6), and the explicit opt-out (`http.filter.ipaddress.exclude: ""`) is asserted to install no rules. The integration check is `HttpProtocolIPFilterTest`: fetching `http://localhost:<port>` (resolves to loopback) under the shipped list throws the interceptor's `Forbidden connection to IP address …` IOException, and the same fetch returns 200 once the exclude list is overridden to empty — so the policy is shown enforced by the real HTTP protocol path, not just the predicates.
Full core suite green (441). The workflow run will need a maintainer's workflow approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-5559804635
@abhinav-phi CI is red here, could you have a look please?
Two Checkstyle violations, both the import order I flagged in the review:
```
core/src/test/java/org/apache/stormcrawler/filtering/DefaultRegexFiltersPrivateRangeTest.java:[26,1]
CustomImportOrder: 'java.net.MalformedURLException' should be before 'java.nio.charset.StandardCharsets'
core/src/test/java/org/apache/stormcrawler/filtering/DefaultRegexFiltersPrivateRangeTest.java:[27,1]
CustomImportOrder: 'java.net.URL' should be before 'java.nio.charset.StandardCharsets'
```
Reproduce with `mvn -B -Prat -DskipTests verify -Dskip.format.code=false`, fix with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`.
[Run 33998813190](https://github.com/apache/stormcrawler/actions/runs/33998813190)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#discussion_r3944167855
##########
archetype/src/main/resources/archetype-resources/src/main/resources/default-regex-filters.txt:
##########
@@ -27,6 +39,14 @@
-^https?://(?:192\.168(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
# 172.16.0.0/12
-^https?://(?:172\.(?:1[6789]|2[0-9]|3[01])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
Review Comment:
`100.64.0.0/10` is `100.64.0.0` to `100.127.255.255`, i.e. second octet 64 to 127.
The alternation is `6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7]`, which blocks 60 to 64 (60 to 63 are public space) and **misses 65 to 69 entirely**. The test only checks `100.64.0.1` and `100.127.255.254`, so neither end of the mistake is covered.
```suggestion
-^https?://(?:100\.(?:6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
```
Please apply to all four copies of the file and add `100.65.0.1` and `100.63.0.1` to `otherNonRoutableRangesAreRejected`.
##########
core/src/main/resources/crawler-default.yaml:
##########
@@ -161,8 +161,12 @@ config:
# - "linklocal" (matches InetAddress.isLinkLocalAddress())
# Only addresses matching an include rule are fetched (empty means all are
# allowed), addresses matching an exclude rule are always blocked.
- # http.filter.ipaddress.include:
- # http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal"
+ # The exclude list is enabled by default: a fetched page decides which hosts
+ # the fetcher connects to, and loopback, private and link-local ranges host
+ # unauthenticated services (e.g. cloud instance metadata) which a public
+ # index must not leak into. Set http.filter.ipaddress.include to any value
Review Comment:
This advice does not work as written.
> Set http.filter.ipaddress.include to any value (e.g. "0.0.0.0/0") and leave the exclude list empty
Exclude wins over include in `IPFilterRules.accept`, so setting only `include` changes nothing: the shipped exclude list on line 169 still applies. The operator has to write `http.filter.ipaddress.exclude: ""` explicitly. Please say that.
Also, `fd00::/8` here only covers the locally-assigned half of ULA; `fc00::/7` is the full range. Same point as in #2119.
##########
archetype/src/main/resources/archetype-resources/src/main/resources/default-regex-filters.txt:
##########
@@ -14,11 +14,23 @@
# of services running on the crawling machine (e.g., Elasticsearch,
# Storm)
#
-# - exclude localhost and loop-back addresses
+# NOTE: these rules judge the bytes of the URL only. A host name which
+# resolves into one of the ranges below is not caught here; the IP
+# address filter (http.filter.ipaddress.exclude) sees the address the
+# fetcher actually connects to and is the authoritative check.
+#
+# - exclude localhost and loop-back addresses, including the abbreviated
+# and integer IPv4 forms the resolver maps to 127.0.0.1
# http://localhost:8080
# http://127.0.0.1/ .. http://127.255.255.255/
+# http://127.1/
+# http://2130706433/
# http://[::1]/
--^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3}|\[::1\])(?::\d+)?(?:/|$)
+-^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){0,3}|\[::1\])(?::\d+)?(?:/|$)
Review Comment:
Case-sensitive, so `LOCALHOST` passes. Worth `(?i)` here too, same as the IPv6 rule on line 49.
##########
archetype/src/main/resources/archetype-resources/src/main/resources/default-regex-filters.txt:
##########
@@ -27,6 +39,14 @@
-^https?://(?:192\.168(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
# 172.16.0.0/12
-^https?://(?:172\.(?:1[6789]|2[0-9]|3[01])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 0.0.0.0/8 ("this network")
+-^https?://(?:0(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){1,3})(?::\d+)?(?:/|$)
+# IPv6 unique-local (fc00::/7) and link-local (fe80::/10) addresses
+-^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)
Review Comment:
Two gaps here.
**Case.** `RegexURLFilter` does not compile with `CASE_INSENSITIVE` (the suffix rule at the top of the file carries its own `(?i)`), so `[FD00::1]` and `[FE80::1]` pass. Same for `LOCALHOST` on line 29.
**Loopback spellings.** Both of these reach loopback through the JVM resolver while matching neither rule:
```
InetAddress.getByName("::ffff:127.0.0.1") -> /127.0.0.1
InetAddress.getByName("0:0:0:0:0:0:0:1") -> /0:0:0:0:0:0:0:1 (== ::1)
```
```suggestion
-^(?i)https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)
# loopback reached through an IPv4-mapped or fully expanded IPv6 literal
-^(?i)https?://\[(?:::ffff:127\.|(?:0*:){7}0*1\])
```
`isLoopbackAddress()` catches all of these after DNS now that the IP filter is on by default, so this is defence in depth, but catching them earlier is what this file is for.
##########
core/src/test/resources/default-regex-filters.txt:
##########
@@ -3,11 +3,50 @@
# skip image and other suffixes we can't parse or are not likely to be relevant
# if you want to crawl images or videos or archives then you should comment out this line
--(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|zip|gz|rpm|tgz|mov|exe|jpeg|bmp|js|mpg|mp3|mp4)(\?|&|$)
+-(?i)\.(apk|deb|cab|iso|gif|jpg|png|svg|ico|css|sit|eps|wmf|rar|tar|jar|zip|gz|bz2|rpm|tgz|mov|exe|jpeg|jpe|bmp|js|mpg|mp3|mp4|m4a|ogv|kml|wmv|swf|flv|mkv|m4v|webm|ra|wma|wav|avi|xspf|m3u)(\?|&|$)
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-# very time-consuming : use only if necessary
+# very time-consuming : use BasicURLFilter instead
# -.*(/[^/]+)/[^/]+\1/[^/]+\1/
+# exclude localhost and equivalents to avoid that information
+# can be leaked by placing faked links pointing to web interfaces
+# of services running on the crawling machine (e.g., Elasticsearch,
+# Storm)
+#
+# NOTE: these rules judge the bytes of the URL only. A host name which
+# resolves into one of the ranges below is not caught here; the IP
+# address filter (http.filter.ipaddress.exclude) sees the address the
+# fetcher actually connects to and is the authoritative check.
+#
+# - exclude localhost and loop-back addresses, including the abbreviated
+# and integer IPv4 forms the resolver maps to 127.0.0.1
+# http://localhost:8080
+# http://127.0.0.1/ .. http://127.255.255.255/
+# http://127.1/
+# http://2130706433/
+# http://[::1]/
+-^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){0,3}|\[::1\])(?::\d+)?(?:/|$)
+# the integer form of an IPv4 address resolves to the address it encodes
+# (e.g. http://2130706433/ is 127.0.0.1); such URLs are blocked outright,
+# a crawl that needs one can rewrite it to its dotted form
+-^https?://\d{1,10}(?::\d+)?(?:/|$)
+#
+# - exclude private IP address spaces
+# 10.0.0.0/8
+-^https?://(?:10(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3})(?::\d+)?(?:/|$)
+# 192.168.0.0/16
+-^https?://(?:192\.168(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 172.16.0.0/12
+-^https?://(?:172\.(?:1[6789]|2[0-9]|3[01])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
Review Comment:
Same CGNAT bug as in the archetype copy; see the comment there.
##########
core/src/test/resources/default-regex-filters.txt:
##########
@@ -3,11 +3,50 @@
# skip image and other suffixes we can't parse or are not likely to be relevant
# if you want to crawl images or videos or archives then you should comment out this line
--(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|zip|gz|rpm|tgz|mov|exe|jpeg|bmp|js|mpg|mp3|mp4)(\?|&|$)
+-(?i)\.(apk|deb|cab|iso|gif|jpg|png|svg|ico|css|sit|eps|wmf|rar|tar|jar|zip|gz|bz2|rpm|tgz|mov|exe|jpeg|jpe|bmp|js|mpg|mp3|mp4|m4a|ogv|kml|wmv|swf|flv|mkv|m4v|webm|ra|wma|wav|avi|xspf|m3u)(\?|&|$)
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-# very time-consuming : use only if necessary
+# very time-consuming : use BasicURLFilter instead
# -.*(/[^/]+)/[^/]+\1/[^/]+\1/
+# exclude localhost and equivalents to avoid that information
+# can be leaked by placing faked links pointing to web interfaces
+# of services running on the crawling machine (e.g., Elasticsearch,
+# Storm)
+#
+# NOTE: these rules judge the bytes of the URL only. A host name which
+# resolves into one of the ranges below is not caught here; the IP
+# address filter (http.filter.ipaddress.exclude) sees the address the
+# fetcher actually connects to and is the authoritative check.
+#
+# - exclude localhost and loop-back addresses, including the abbreviated
+# and integer IPv4 forms the resolver maps to 127.0.0.1
+# http://localhost:8080
+# http://127.0.0.1/ .. http://127.255.255.255/
+# http://127.1/
+# http://2130706433/
+# http://[::1]/
+-^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){0,3}|\[::1\])(?::\d+)?(?:/|$)
+# the integer form of an IPv4 address resolves to the address it encodes
+# (e.g. http://2130706433/ is 127.0.0.1); such URLs are blocked outright,
+# a crawl that needs one can rewrite it to its dotted form
+-^https?://\d{1,10}(?::\d+)?(?:/|$)
+#
+# - exclude private IP address spaces
+# 10.0.0.0/8
+-^https?://(?:10(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3})(?::\d+)?(?:/|$)
+# 192.168.0.0/16
+-^https?://(?:192\.168(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 172.16.0.0/12
+-^https?://(?:172\.(?:1[6789]|2[0-9]|3[01])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+# 0.0.0.0/8 ("this network")
+-^https?://(?:0(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){1,3})(?::\d+)?(?:/|$)
+# IPv6 unique-local (fc00::/7) and link-local (fe80::/10) addresses
+-^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)
Review Comment:
Same IPv6 gaps as in the archetype copy; see the comment there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#discussion_r3976885444
##########
core/src/main/resources/crawler-default.yaml:
##########
@@ -165,8 +165,14 @@ config:
# topology from starting.
# Only addresses matching an include rule are fetched (empty means all are
# allowed), addresses matching an exclude rule are always blocked.
- # http.filter.ipaddress.include:
- # http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,fc00::/7"
+ # The exclude list is enabled by default: a fetched page decides which hosts
+ # the fetcher connects to, and loopback, private and link-local ranges host
+ # unauthenticated services (e.g. cloud instance metadata) which a public
+ # index must not leak into. To crawl an intranet, override the exclude list
+ # explicitly (exclude rules win over include rules, so setting only
+ # http.filter.ipaddress.include does not undo this default), e.g.
+ # http.filter.ipaddress.exclude: ""
+ http.filter.ipaddress.exclude: "localhost,sitelocal,linklocal,anylocal,multicast,100.64.0.0/10,0.0.0.0/8,fc00::/7,::/128"
Review Comment:
Fixed in ed35b139, taking the first option. The interceptor now lets a connection through when `route().proxy()` is not `DIRECT`: the proxy resolves the target itself, so the only address the filter ever saw was the proxy's. A log line at configure time says that the filter does not apply to proxied fetches, and the yaml and configuration.adoc document that the proxy's own egress rules decide there. Pinned by `fetchThroughProxyOnLoopbackIsNotFiltered`, which fetches through a proxy on 127.0.0.1 under the shipped exclude list and fails on the previous code with "Forbidden connection to IP address 127.0.0.1".
The same commit updates configuration.adoc for your other point: the table shows the new default of `http.filter.ipaddress.exclude`, and the IP Address Filtering section no longer calls the filter optional and explains how to override it for an intranet crawl.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-5580731563
CI is green on the latest run (rat + build, run for commit 775d47d2). Formatting-only follow-up commit; no code changes since the review fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Keep the fetcher off private address space by default [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on PR #2127:
URL: https://github.com/apache/stormcrawler/pull/2127#issuecomment-5574422474
Thanks — pushed 55395054 closing everything:
- **CGNAT regex:** fixed to `6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7]` in the archetype and both external copies; `100.65.0.1` (blocked) and `100.63.0.1` (accepted) are in the tests, both ends of the mistake.
- **Case:** the private-range rules now carry `(?i)` — `LOCALHOST`, `[FD00::1]`, `[FE80::1]` are pinned by `caseInsensitiveSpellingsAreRejected`.
- **Loopback via IPv6 literals:** added your suggested rule for `::ffff:127.0.0.1` and the fully-expanded `(0*:){7}0*1` form, pinned by tests.
- **Unrelated change:** reverted `core/src/test/resources/default-regex-filters.txt` to its upstream content. The archetype rules ship as `default-regex-filters-archetype.txt` for the new test instead, so `RegexFilterTest`/`URLFilterBoltTest` are untouched.
- **Yaml advice:** corrected — the comment now says exclude rules win over include rules and the operator has to override `http.filter.ipaddress.exclude` explicitly (e.g. `""`). Also took the opportunity to fix the shipped value to `fc00::/7` (full ULA), consistent with what #2119 is doing to the example.
- **Resolver notes:** put into the description — hex is rejected (`UnknownHostException`) and leading zeros are not octal (`0177.0.0.1` → `177.0.0.1`, public), so the `\d{1,10}` rule closes the only numeric form that works; a test documents both.
Overlapping with #2119: #2127 owns the *shipped value* of `http.filter.ipaddress.exclude` (it flips it from commented to enabled); #2119 only updates the commented example and the keyword docs. #2119 merges first and I'll rebase this one if the lines collide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org