You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "jnioche (via GitHub)" <gi...@apache.org> on 2023/11/06 14:18:13 UTC
[PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
jnioche opened a new pull request, #796:
URL: https://github.com/apache/nutch/pull/796
Checks the length of the query and path elements of a URL.
see https://issues.apache.org/jira/browse/NUTCH-3025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on code in PR #796:
URL: https://github.com/apache/nutch/pull/796#discussion_r1384536930
##########
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java:
##########
@@ -97,9 +97,17 @@ public class FastURLFilter implements URLFilter {
private Configuration conf;
public static final String URLFILTER_FAST_FILE = "urlfilter.fast.file";
+ public static final String URLFILTER_FAST_PATH_MAX_LENGTH = "urlfilter.fast.url.path.max.length";
+ public static final String URLFILTER_FAST_QUERY_MAX_LENGTH = "urlfilter.fast.url.query.max.length";
+
Review Comment:
What about adding a third limit for path and query combined ([URL.getFile()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/URL.html#getFile())?
- if somebody defined two generous but reasonable limits (for example, 2048) for both path and query, the resulting URL may still get quite long and cause troubles
- also the HTTP GET request includes both path and query
- for many use cases it should sufficient to just set this limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel merged PR #796:
URL: https://github.com/apache/nutch/pull/796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #796:
URL: https://github.com/apache/nutch/pull/796#issuecomment-1798221743
Writing a test for this thing is an absolute pain. The way the filters are used for real is that their method setConf is called and the rules are loaded using _getConfResourceAsReader_, i.e. they are expected to be in the jar.
The tests do not rely on that mechanism and instead instantiate the filter with the reader for its rules. This means that the conf is not used at all and therefore we can't use that to load the value for the length based filters. I will add another constructor with the reader + conf so that we can test based on the length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on code in PR #796:
URL: https://github.com/apache/nutch/pull/796#discussion_r1384621727
##########
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java:
##########
@@ -97,9 +97,17 @@ public class FastURLFilter implements URLFilter {
private Configuration conf;
public static final String URLFILTER_FAST_FILE = "urlfilter.fast.file";
+ public static final String URLFILTER_FAST_PATH_MAX_LENGTH = "urlfilter.fast.url.path.max.length";
+ public static final String URLFILTER_FAST_QUERY_MAX_LENGTH = "urlfilter.fast.url.query.max.length";
+
Review Comment:
I might keep things simple and just add a size limit on the whole URL regardless of its parts, similar to [what is done in StormCrawler.](https://github.com/DigitalPebble/storm-crawler/blob/ef31e509139cccb2919c345ef343c4fcfb2f1ec5/core/src/main/java/com/digitalpebble/stormcrawler/filtering/basic/BasicURLFilter.java#L30C17-L30C26)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #796:
URL: https://github.com/apache/nutch/pull/796#issuecomment-1801938355
@sebastian-nagel merged the changes from master and made a few improvements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3025] urlfilter-fast to filter based on the length of the URL [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on PR #796:
URL: https://github.com/apache/nutch/pull/796#issuecomment-1802531264
Thanks, @jnioche!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org