[I] IPFilterRules has no keyword for link-local addresses and the documented example leaves them allowed [stormcrawler]

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 opened a new issue, #2086:
URL: https://github.com/apache/stormcrawler/issues/2086

   
   ## What happens
   `IPFilterRules.parseIPRules()` understands three keywords: `localhost` and `loopback` map to `InetAddress::isLoopbackAddress`, and `sitelocal` maps to `InetAddress::isSiteLocalAddress`. For IPv4, `isSiteLocalAddress` covers 10/8, 172.16/12 and 192.168/16 only. Addresses in 169.254.0.0/16, 100.64.0.0/10 and 0.0.0.0/8, and IPv6 addresses in fd00::/8 and fe80::/10, match none of the keywords, so the example shipped in `crawler-default.yaml`, `"localhost,sitelocal"`, still allows them. Anything else is parsed as a CIDR block or a single address, and a value that parses as neither is dropped with a `LOG.error`, so a typo in an exclude list removes a rule the operator thought was in force.
   
   ## Where
   `core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java:111-129`, keys `http.filter.ipaddress.include` and `http.filter.ipaddress.exclude`. The example is at `core/src/main/resources/crawler-default.yaml:162` and the same keyword list is in the class javadoc at lines 45-52.
   
   ```java
   case "sitelocal":
       rules.add(InetAddress::isSiteLocalAddress);
       break;
   ```
   
   ## Why it matters
   An operator who turns the filter on and copies the shipped example gets loopback and RFC1918 blocked and nothing else. On a cloud worker the link-local range is the one that carries an unauthenticated instance metadata service, and it is reachable from the worker whatever DNS name the crawled link used, because the check runs on the resolved socket address. The ranges can be written today as plain CIDR rules, so this is about the vocabulary and the example being incomplete rather than a missing capability. The interceptor is also skipped when a proxy is configured, as its javadoc notes.
   
   ## Reproduction
   
   Save as `core/src/test/java/org/apache/stormcrawler/protocol/IPFilterRulesKeywordsTest.java`.
   
   ```java
   /*
    * Licensed to the Apache Software Foundation (ASF) under one or more
    * contributor license agreements.  See the NOTICE file distributed with
    * this work for additional information regarding copyright ownership.
    * The ASF licenses this file to you under the Apache License, Version 2.0
    * (the "License"); you may not use this file except in compliance with
    * the License.  You may obtain a copy of the License at
    *
    *      http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing, software
    * distributed under the License is distributed on an "AS IS" BASIS,
    * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    * See the License for the specific language governing permissions and
    * limitations under the License.
    */
   
   package org.apache.stormcrawler.protocol;
   
   import static org.junit.jupiter.api.Assertions.assertFalse;
   
   import com.google.common.net.InetAddresses;
   import java.net.InetAddress;
   import java.util.HashMap;
   import java.util.Map;
   import org.junit.jupiter.api.Test;
   
   /** The rule keywords and the example shipped in crawler-default.yaml. */
   class IPFilterRulesKeywordsTest {
   
       private static InetAddress ip(String address) {
           return InetAddresses.forString(address);
       }
   
       private static IPFilterRules exclude(Object exclude) {
           Map<String, Object> conf = new HashMap<>();
           conf.put(IPFilterRules.EXCLUDE_RULES_KEY, exclude);
           return new IPFilterRules(conf);
       }
   
       /** The example from crawler-default.yaml. */
       @Test
       void documentedExampleBlocksTheNonRoutableRanges() {
           IPFilterRules r = exclude("localhost,sitelocal");
           assertFalse(r.accept(ip("169.254.169.254")), "IPv4 link-local");
           assertFalse(r.accept(ip("100.64.0.1")), "CGNAT");
           assertFalse(r.accept(ip("fd00::1")), "IPv6 unique local");
           assertFalse(r.accept(ip("fe80::1")), "IPv6 link-local");
       }
   
       /** There is no keyword for link-local addresses. */
       @Test
       void linkLocalKeywordIsSupported() {
           IPFilterRules r = exclude("linklocal");
           assertFalse(r.accept(ip("169.254.169.254")));
           assertFalse(r.accept(ip("fe80::1")));
       }
   }
   ```
   
   Run it:
   
   ```
   mvn -pl core test -Dtest=IPFilterRulesKeywordsTest
   ```
   
   Both tests fail on main. It sits next to the existing `IPFilterRulesTest`.
   
   ```
   [main] ERROR org.apache.stormcrawler.protocol.IPFilterRules - Failed to parse linklocal as CIDR,
       ignoring it while configuring IP rules (http.filter.ipaddress.exclude)
   [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
   [ERROR]   IPFilterRulesKeywordsTest.documentedExampleBlocksTheNonRoutableRanges:45
       IPv4 link-local ==> expected: <false> but was: <true>
   [ERROR]   IPFilterRulesKeywordsTest.linkLocalKeywordIsSupported:55 expected: <false> but was: <true>
   ```
   
   ## Suggested fix
   Add a `linklocal` keyword mapping to `InetAddress::isLinkLocalAddress` in `IPFilterRules.parseIPRules`, and consider `anylocal` and `multicast` alongside it. Update the class javadoc and the commented example in `crawler-default.yaml` to `"loopback,sitelocal,linklocal,100.64.0.0/10,fd00::/8"`. Change the unparseable-rule branch to throw instead of logging, so a mistyped exclude rule stops the topology rather than silently widening what it will connect to; that is a behaviour change for anyone who has a bad rule in their config today, and the existing `invalidRuleIsIgnored` test in `IPFilterRulesTest` needs to be updated with it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] IPFilterRules has no keyword for link-local addresses and the documented example leaves them allowed [stormcrawler]

["abhinav-phi (via GitHub)" <gi...@apache.org>]

abhinav-phi commented on issue #2086:
URL: https://github.com/apache/stormcrawler/issues/2086#issuecomment-5467108399

   Hi! I'd like to work on this.
   Plan: I'll add a keyword for link-local addresses to IPFilterRules, fix the documented example so it doesn't leave those addresses allowed, and add tests for link-local ranges.
   /assign take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] IPFilterRules has no keyword for link-local addresses and the documented example leaves them allowed [stormcrawler]

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 closed issue #2086: IPFilterRules has no keyword for link-local addresses and the documented example leaves them allowed
URL: https://github.com/apache/stormcrawler/issues/2086


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org