You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/08/31 12:15:09 UTC
[jira] [Created] (NUTCH-1101) Options to purge db_gone records in
updatedb
Options to purge db_gone records in updatedb
--------------------------------------------
Key: NUTCH-1101
URL: https://issues.apache.org/jira/browse/NUTCH-1101
Project: Nutch
Issue Type: New Feature
Components: linkdb
Affects Versions: 1.4
Reporter: Markus Jelsma
Fix For: 1.4
Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1101:
---------------------------------
Patch Info: [Patch Available]
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094431#comment-13094431 ]
Markus Jelsma commented on NUTCH-1101:
--------------------------------------
Please comment on implemenation and usefulness and on the fact that it only deletes existing records.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094436#comment-13094436 ]
Markus Jelsma commented on NUTCH-1101:
--------------------------------------
It's been tested on the production crawl db that actually suffered with this problem. The output of two readdb -stats jobs all add up and make sense. The records are purged and new DB_GONE records have been added from the segment. These are to be removed in the next update cycle.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097928#comment-13097928 ]
Markus Jelsma commented on NUTCH-1101:
--------------------------------------
Comments? I'd like to send this one in.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1101:
---------------------------------
Attachment: NUTCH-1101-1.4-1.patch
Patch for 1.4. This patch adds a -purge404 switch to the updatedb command and, if enabled, removes all _existing_ 404 records (STATUS_DB_GONE) from the crawl db. It does not remove new 404 records (STATUS_FETCH_GONE) being added from the segment.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1101:
---------------------------------
Attachment: NUTCH-1101-1.4-2.patch
Thanks Julien! I've modified the stuff to rely on the config option and ability to use Tool to set the option on the CLI. Tested and confirmed to work.
Also added setting to nutch-default.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch, NUTCH-1101-1.4-2.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1101.
--------------------------------
Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch, NUTCH-1101-1.4-2.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1101.
----------------------------------
Resolution: Fixed
Committed for 1.4 in rev.1167096.
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch, NUTCH-1101-1.4-2.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097955#comment-13097955 ]
Julien Nioche commented on NUTCH-1101:
--------------------------------------
The functionality makes sense but I am not really OK with the change of method signature
{code}
public void update(Path crawlDb, Path[] segments, boolean normalize, boolean filter, boolean url404Purging)
{code}
and don't think we need to add yet another parameter to the class. Instead we could just rely on the configuration files - which is exactly what you were advocating for in the case of NUTCH-1102 :-)
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (NUTCH-1101) Options to purge db_gone records in
updatedb
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma reassigned NUTCH-1101:
------------------------------------
Assignee: Markus Jelsma
> Options to purge db_gone records in updatedb
> --------------------------------------------
>
> Key: NUTCH-1101
> URL: https://issues.apache.org/jira/browse/NUTCH-1101
> Project: Nutch
> Issue Type: New Feature
> Components: linkdb
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1101-1.4-1.patch
>
>
> Add option to updatedb to filter out records with status db_gone (http 404). This is especially useful in cases where a crawl db is targetted at only a specific site. If the site, for some reason, suddenly changes a lot of url's we'll get a crawl db filled with garbage. Since the targetted site is known (or controlled) it is safe to get rid of all these url's: reduce db size, reduce useless http requests.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira