You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Emmanuel Joke (JIRA)" <ji...@apache.org> on 2007/05/22 09:38:16 UTC
[jira] Created: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Avoid parsing uneccessary links and get a more relevant outlink list
--------------------------------------------------------------------
Key: NUTCH-488
URL: https://issues.apache.org/jira/browse/NUTCH-488
Project: Nutch
Issue Type: Improvement
Affects Versions: 0.9.0
Environment: Windows, Java 1.5
Reporter: Emmanuel Joke
NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Posted by "Emmanuel Joke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Joke updated NUTCH-488:
--------------------------------
Attachment: nutch-default.xml.patch
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis Kubes closed NUTCH-488.
------------------------------
Closed
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Assignee: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535773 ]
Dennis Kubes commented on NUTCH-488:
------------------------------------
If no one has any objections. This is working in a production environment and I think it is a very useful contribution, I am going to commit this.
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535014 ]
Dennis Kubes commented on NUTCH-488:
------------------------------------
Tested. Working good +1
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Posted by "Emmanuel Joke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Joke updated NUTCH-488:
--------------------------------
Attachment: DOMContentUtils.patch
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529382 ]
Doğacan Güney commented on NUTCH-488:
-------------------------------------
Looks good but a couple of comments:
* You should use conf.getStrings instead of splitting value yourself
* This line:
if ( ! ignoreTags.contains("form") && conf.getBoolean("parser.html.form.use_action", false))
should perhaps contain an OR to keep backward compatibility
* doing the "if ( !ignoreTags )" for everything is error prone. Perhaps we can add a Map<tag-name, tag-attr> type of thing so:
for (Entry<String, String> entry: ignoreMap.entrySet()) {
String tag = entry.getKey();
String attr = entry.getValue();
linkParams.put(tag, new LinkParams(tag, attr, 0));
}
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535855 ]
Doğacan Güney commented on NUTCH-488:
-------------------------------------
I haven't tested it but the code looks good and clean.
+1 from me.
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis Kubes resolved NUTCH-488.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Assignee: Emmanuel Joke
This has been committed. Thanks Marcin and Emmanuel.
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Assignee: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Posted by "Marcin Okraszewski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcin Okraszewski updated NUTCH-488:
-------------------------------------
Attachment: ignore_tags_v3.patch
OK, yet another approach based on Doğacan comments. Sorry for delay, but I didn't notice the comment earlier.
- I didn't notice the conf.getStrings() method. Thanks for hint :)
- I did made the backward compatibility with the "use_action" param, but it works a bit different now, if there is no value set. Now, default is that it should use the forms. But it can be dropped with ignore_tags setting if not specified. If someone has the use_action set to true explicite, then it cannot be overridden by the ignore_tags. It is still a bit inconsitent, but it is understandable that specific setting (use_action) has precedence. If default is "false" then if you do not have "use_action" defined and not added to ignore_tags, then one could expect that form is taken. But it wouldn't be. Keeping the backward compatibility make the code a bit clumsy :( ... and I think I've made it over flexible, but that was the cleanest solution here.
- For the repeating if; I agree, it is error prone, but on the other hand it is easy to understand. I didn't quite understand Dogacan's proposal :( but I think I did something acceptable - simply remove all specified tags from link params.
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535875 ]
Andrzej Bialecki commented on NUTCH-488:
-----------------------------------------
+1. The nutch-default description has a typo, instead of "coma" it should be "comma".
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-488) Avoid parsing uneccessary links and
get a more relevant outlink list
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536102 ]
Hudson commented on NUTCH-488:
------------------------------
Integrated in Nutch-Nightly #241 (See [http://lucene.zones.apache.org:8080/hudson/job/Nutch-Nightly/241/])
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Assignee: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, ignore_tags_v3.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-488) Avoid parsing uneccessary links and get
a more relevant outlink list
Posted by "Marcin Okraszewski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcin Okraszewski updated NUTCH-488:
-------------------------------------
Attachment: ignore_tags_v2.patch
Yet another patch. The differences to the Emmanuel's patch are:
- there is a single property to manage which links should be taken; simply coma-separated list of tags to ignore.
- all tags can be blocked, including "a". Emmanuel's proposition do not allow to turn off a,area,frame,iframe
- the coma-separated property seems to be more flexible once a new tag was added...
The patch was done against trunk. I hope to see it in Nutch 1.0 :)
> Avoid parsing uneccessary links and get a more relevant outlink list
> --------------------------------------------------------------------
>
> Key: NUTCH-488
> URL: https://issues.apache.org/jira/browse/NUTCH-488
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: Windows, Java 1.5
> Reporter: Emmanuel Joke
> Attachments: DOMContentUtils.patch, ignore_tags_v2.patch, nutch-default.xml.patch
>
>
> NekoHTML parser use a method to extract all outlinks from the HTML page. It will extracts them from the HTML content based on the list of param defined in the method setConf(). Then this list of links will be truncated to be limit to the the maximum number of outlinks that we'll process for a page defined in nutch-default.xml (db.max.outlinks.per.page = 100 by default ) and finally it will be go through all urlfilter defined.
> Unfortunetly it can happen that the list of outlinks is more than 100, so it will truncated the list and could remove some relevant links.
> So I've added few options in the nutch-default.xml in order to enable/disable the extraction of specific HTML Tag links in this parser (SCRIPT, IMG, FORM, LINK).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.