[jira] Resolved: (SOLR-799) Add support for hash based exact/near duplicate document handling

["Yonik Seeley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley resolved SOLR-799.
-------------------------------

       Resolution: Fixed
    Fix Version/s: 1.4

- fixed so that all values of a multi-valued field are included in the hash
- changed so that no string additions are done for performance
- moved HEX_CHARS to StrUtils
- changed "fields" to be a comma separated list (per the wiki documentation... this may be more consistent if we allow this to be specified as a request parameter later, but it's subjective for sure.  we could always add support for both arrays and comma separated lists).
- changed the hashcode generation to work with any sized hash (was previously hardcoded to 16 bytes)
- added lookup3ycs http://yonik.wordpress.com/2008/06/14/lookup3ycs-a-standard-high-performance-string-hash/   lookup3ycs can do hashes directly on strings (no need to convert to bytes first).  I used the 64 bit variant, which is more than enough to prevent false collisions, and it resulted in a 27% speedup in total indexing time (after removing other cruft from the schema such as copyFields and default values).
- tested with 10M documents to verify that no collisions occur with both MD5 and lookup3
- Committed!  Thanks Mark!  And thanks to everyone else for the great feedback.

> Add support for hash based exact/near duplicate document handling
> -----------------------------------------------------------------
>
>                 Key: SOLR-799
>                 URL: https://issues.apache.org/jira/browse/SOLR-799
>             Project: Solr
>          Issue Type: New Feature
>          Components: update
>            Reporter: Mark Miller
>            Assignee: Yonik Seeley
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-799.patch, SOLR-799.patch, SOLR-799.patch, SOLR-799.patch
>
>
> Hash based duplicate document detection is efficient and allows for blocking as well as field collapsing. Lets put it into solr. 
> http://wiki.apache.org/solr/Deduplication

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.