You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Julien Nioche (JIRA)" <ji...@apache.org> on 2011/07/12 19:26:59 UTC
[jira] [Created] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Add pattern for filtering .js in default url filters
----------------------------------------------------
Key: NUTCH-1043
URL: https://issues.apache.org/jira/browse/NUTCH-1043
Project: Nutch
Issue Type: Task
Affects Versions: 1.4, 2.0
Reporter: Julien Nioche
Priority: Minor
Fix For: 1.4, 2.0
The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1043:
---------------------------------
Patch Info: [Patch Available]
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche resolved NUTCH-1043.
----------------------------------
Resolution: Fixed
Committed revision 1147796 -> 1.4
Committed revision 1147798 -> 2.0 (trunk)
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067481#comment-13067481 ]
Hudson commented on NUTCH-1043:
-------------------------------
Integrated in Nutch-trunk #1550 (See [https://builds.apache.org/job/Nutch-trunk/1550/])
NUTCH-1043 Add pattern for filtering .js in default url filters
jnioche : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1147798
Files :
* /nutch/trunk/conf/automaton-urlfilter.txt.template
* /nutch/trunk/conf/regex-urlfilter.txt.template
* /nutch/trunk/CHANGES.txt
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064095#comment-13064095 ]
Markus Jelsma commented on NUTCH-1043:
--------------------------------------
Good reasoning, the suffix url filter is more suitable for dealing with large amounts of prohibited suffixes. Maybe only suffixes of formats that are known to be large files (tif, ram etc) should be added to the default regex filter as to prevent users from accidentally downloading many MiB's of unwanted (and unparsable) data.
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064065#comment-13064065 ]
Julien Nioche commented on NUTCH-1043:
--------------------------------------
My point here is not to add all the suffixes that are not supported by the default configuration of parsers but only to add the most common ones to the url filters, like what we currently have for the existing URL filters i.e.
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
@Markus -> the suffix urlfilter already covers some of the things that you've listed. You're not using it? It is not activated by default but people would be able to do so if they need the extended list of suffixes
I am not against adding more suffixes to the URL filters but would rather keep it as simple and close to the existing as possible. What do you think?
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1043.
--------------------------------
Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, nutchgora
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, nutchgora
>
> Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064039#comment-13064039 ]
Markus Jelsma commented on NUTCH-1043:
--------------------------------------
Great but how about changing this issue to support a wider range of suffixes? I've got plenty of suffixes that are not being parsed by any parser nor Tika such as kss, cur, wax, ram, wma etc.
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche updated NUTCH-1043:
---------------------------------
Attachment: NUTCH-1043.patch
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1043.patch
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1043) Add pattern for filtering .js in
default url filters
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064045#comment-13064045 ]
Lewis John McGibbney commented on NUTCH-1043:
---------------------------------------------
I think some discussion on this topic could save us experiencing a substantial Gotcha issue in the future. Personally I am not dealing with the number of suffiixes mentioned above, however this means little overall and I personally back getting something out in the open which addresses this issue.
> Add pattern for filtering .js in default url filters
> ----------------------------------------------------
>
> Key: NUTCH-1043
> URL: https://issues.apache.org/jira/browse/NUTCH-1043
> Project: Nutch
> Issue Type: Task
> Affects Versions: 1.4, 2.0
> Reporter: Julien Nioche
> Priority: Minor
> Fix For: 1.4, 2.0
>
>
> The Javascript parser is not used by default as it is extremely noisy, however the default URL filters do not filter out URLs ending in .js and the default parser (Tika) can't parse them. In a nutshell we are fetching URLS that we know can't be parsed.
> I suggest that we add a regex to the default URL filters. If people are interested in fetching and parsing .js files they can activate the plugin in their conf and remove the regex in the URL filters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira