You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Apache Wiki <wi...@apache.org> on 2012/06/20 15:06:19 UTC
[Nutch Wiki] Update of "DebugTool" by LewisJohnMcgibbney
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "DebugTool" page has been changed by LewisJohnMcgibbney:
http://wiki.apache.org/nutch/DebugTool?action=diff&rev1=4&rev2=5
It should be possible to generate information that would have answered all of the "is it X" questions that came up during a user's crawl. E.g.
1. which URLs were put on the fetch list, versus skipped.
- 1. which fetched documents were truncated.
+ 1. which fetched documents were truncated.
+ - The code currently has primitive logging for all parse plugins to log verification of truncation to stdout. What more could we do here? It is a common problem so would be good to improve this area.
1. which URLs in a parsed page were skipped, due to the max outlinks per page limit.
1. which URLs got filtered by regex, prefix, suffix, domain filters
1. exclusions by robots directives