You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2016/02/24 14:46:18 UTC

[jira] [Updated] (NUTCH-2232) DeduplicationJob should decode URL's before length is compared

     [ https://issues.apache.org/jira/browse/NUTCH-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma updated NUTCH-2232:
---------------------------------
    Summary: DeduplicationJob should decode URL's before length is compared  (was: DeduplicationJob: Url is not decoded before the url length is compared.)

> DeduplicationJob should decode URL's before length is compared
> --------------------------------------------------------------
>
>                 Key: NUTCH-2232
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2232
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb
>    Affects Versions: 1.11
>            Reporter: Ron van der Vegt
>             Fix For: 1.12
>
>         Attachments: NUTCH-2232.patch
>
>
> When certain documents have the same signature de deduplication script will elect one as duplicate. The urls are stored url encoded in the crawldb. When two urls are compared by url length, the urls are not first decoded. This could lead to misleading url length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)