You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Sebastian Nagel (Jira)" <ji...@apache.org> on 2023/07/20 12:16:00 UTC
[jira] [Commented] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17745061#comment-17745061 ]
Sebastian Nagel commented on NUTCH-2993:
----------------------------------------
Hi [~markus17], the updated patch looks good. I'll open a PR for easier merging. Alternatively, feel free to apply the patch to master and commit it!
> Now the maxDepth resets if it does NOT match/find the pattern.
It sounds plausible to reset the increased depth back to the default value if the URL isn't matched.
Generally, I agree that steering a crawler and keep it focused is always difficult. We also should update the plugin scoring-similarity, not sure whether it's still working with a recent Hadoop and a quite old Mahout version combined. Interestingly, [SimilarityScoringFilter|https://cwiki.apache.org/confluence/display/NUTCH/SimilarityScoringFilter] mentions the filtering of URLs/outlinks as a nice-to-have option.
> ScoringDepth plugin to skip depth check based on URL Pattern
> ------------------------------------------------------------
>
> Key: NUTCH-2993
> URL: https://issues.apache.org/jira/browse/NUTCH-2993
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.20
>
> Attachments: NUTCH-2993-1.15-1.patch, NUTCH-2993-1.15.patch
>
>
> We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites.
> This patch overrides maxDepth for outlinks of URLs matching a configured pattern. URL not matching the pattern get the default max depth value configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)