You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Emmanuel Joke (JIRA)" <ji...@apache.org> on 2007/05/22 10:35:16 UTC
[jira] Created: (NUTCH-489) URLFilter-suffix management of the url
path when the url contains some query parameters
URLFilter-suffix management of the url path when the url contains some query parameters
---------------------------------------------------------------------------------------
Key: NUTCH-489
URL: https://issues.apache.org/jira/browse/NUTCH-489
Project: Nutch
Issue Type: Improvement
Affects Versions: 0.9.0
Environment: Windows, Java 1.5
Reporter: Emmanuel Joke
The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (NUTCH-489) URLFilter-suffix management of the url
path when the url contains some query parameters
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney resolved NUTCH-489.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Assignee: Doğacan Güney
Committed in rev. 548429 with some modifications:
1) Fix minor whitespace issues.
2) Filtering on path is disabled by default. One will have to uncomment line in suffix-urlfilter.txt to enable it.
3) Updated unit test case.
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Assignee: Doğacan Güney
> Fix For: 1.0.0
>
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-489) URLFilter-suffix management of the url
path when the url contains some query parameters
Posted by "Emmanuel Joke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Joke updated NUTCH-489:
--------------------------------
Attachment: SuffixURLFilter.java.patch
suffix-urlfilter.txt.patch
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-489) URLFilter-suffix management of the
url path when the url contains some query parameters
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499777 ]
Doğacan Güney commented on NUTCH-489:
-------------------------------------
Please ignore my last comment. I don't know what I was on when I wrote that. Your patch seems to be correct.
Andrzej, (hope you are reading this:) isn't javadoc in SuffixURLFilter wrong? It says "...., you should use "+.jpg" instead.", but AFAICS, that line would have no effect except possibly switching mode to accept. Since, first character after '+' is not 'I', filter will just skip rest of the line. Am I missing something here?
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-489) URLFilter-suffix management of the
url path when the url contains some query parameters
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498113 ]
Doğacan Güney commented on NUTCH-489:
-------------------------------------
Hmm.. Won't it now cause Nutch to filter on path on a line like this:
-(jpg|JPG)
Because, you just check if line contains a 'P', and line above does.
Perhaps, a better option would be to require P and I be the only characters on their line, like:
+I
+P
# but not +IP
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-489) URLFilter-suffix management of the url
path when the url contains some query parameters
Posted by "Emmanuel Joke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Joke updated NUTCH-489:
--------------------------------
Attachment: SuffixURLFilter_v2.java.patch
My mistake...
I've added a new patchwhich is supposed to:
1- have a correct indentation
2- manage to read the option I and P without any order or constraint.
Hope it will help
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-489) URLFilter-suffix management of the
url path when the url contains some query parameters
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497770 ]
Doğacan Güney commented on NUTCH-489:
-------------------------------------
This is obviously useful but:
* Your patches both in this issue and in NUTCH-488 have whitespace issues. Nutch uses (or at least, tries to use) 2-space indentation. You seem to use a tab-indentation, so your patches end up removing code only to re-add it with wrong indentation.
* I don't understand why new P option only works after I. What if I want this filter to be case-sensitive and to filter on path?
* (Not related to your patch) It seems even a "-I" line makes SuffixURLFilter ignore case. Is that intended?
> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
> Key: NUTCH-489
> URL: https://issues.apache.org/jira/browse/NUTCH-489
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.