You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2016/02/19 12:13:18 UTC
[jira] [Assigned] (NUTCH-2219) Dedup script, allow users to change
the order in which main documents are selected.
[ https://issues.apache.org/jira/browse/NUTCH-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma reassigned NUTCH-2219:
------------------------------------
Assignee: Markus Jelsma
> Dedup script, allow users to change the order in which main documents are selected.
> -----------------------------------------------------------------------------------
>
> Key: NUTCH-2219
> URL: https://issues.apache.org/jira/browse/NUTCH-2219
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb
> Reporter: Ron van der Vegt
> Assignee: Markus Jelsma
> Attachments: NUTCH-2219.patch
>
>
> Current implementation:
> "This command takes a path to a crawldb as parameter and finds duplicates based on the signature. If several entries share the same signature, the one with the highest score is kept. If the scores are the same, then the fetch time is used to determine which one to keep with the most recent one being kept. If their fetch times are the same we keep the one with the shortest URL."
> The order in which the main document is selected is currently not changeable. Therefore I think this option would be nice:
> -compareOrder <score>,<fetchTime>,<url>
> I have written a patch on trunk (rev 1730516). I'm looking forward for any peer review.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)