You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/08/27 12:25:11 UTC

[PR] Honour ignoreOutsideDomain when set on its own in HostURLFilter [stormcrawler]

rzo1 opened a new pull request, #2074:
URL: https://github.com/apache/stormcrawler/pull/2074

   `HostURLFilter.configure()` read the `ignoreOutsideDomain` parameter but tested the `ignoreOutsideHost` node for null. A configuration naming only `ignoreOutsideDomain` was silently discarded and the filter let every URL through; one naming only `ignoreOutsideHost` threw a `NullPointerException` at topology startup. Each parameter is now tested on its own.
   
   Topologies setting only `ignoreOutsideDomain` will start restricting outlinks to the source domain, as configured. Configurations setting both keys, including all three archetypes, are unaffected.
   
   ### For all changes
   
   - [ ] Is there a issue associated with this PR? Is it referenced in the commit message? - no issue
   
   - [ ] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve? - no issue
   
   - [x] Has your PR been rebased against the latest commit within the target branch (typically main)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   - [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`?
   
   ### For code changes
   
   - [ ] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - `core` module tests only (348 run, all green)
   - [x] Have you written or updated unit tests to verify your changes? - two new cases in `HostURLFilterTest`, both fail without the fix
   - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - no new dependencies
   - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a
   - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Honour ignoreOutsideDomain when set on its own in HostURLFilter [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #2074:
URL: https://github.com/apache/stormcrawler/pull/2074#issuecomment-5449810971

   Both made precise: the NPE is now attributed to `ignoreOutsideHost: false`, with a note that `true` short-circuits before the domain parameter is ever read, and the second sentence is scoped to `ignoreOutsideDomain: true`. Commit message and description both updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Honour ignoreOutsideDomain when set on its own in HostURLFilter [stormcrawler]

Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #2074:
URL: https://github.com/apache/stormcrawler/pull/2074#issuecomment-5451624652

   thanks @rzo1 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Honour ignoreOutsideDomain when set on its own in HostURLFilter [stormcrawler]

Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche merged PR #2074:
URL: https://github.com/apache/stormcrawler/pull/2074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org