[jira] [Commented] (SOLR-3473) Distributed deduplication broken

["Chris M. Hostetter (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308892#comment-17308892 ] 

Chris M. Hostetter commented on SOLR-3473:
------------------------------------------

9 years later...

The current state of things, long after the above mentioned related jira's have been addressed, is that SignatureUpdateProcessorFactory *CAN* be safely used in in SolrCloud for two possible usecases:
 * For de-duplication:
 ** the signatureField _MUST_ be the uniqueKey field *AND* the processor _MUST_ be configured to run prior to DistributedUpdateProcessor
 * Solely for generating signatures, w/o de-duplication
 ** overwriteDupes _MUST_ be set to false ... any signatureField may be used, and it may run at any point in the processor chain

If you attempt to use SignatureUpdateProcessorFactory for de-duplication w/ a non-uniqueKey signature field, one of two failure situations are likely to arise:
 * in a multi-shard collection, documents with identical signatureField values will not be removed from any shard (leader) other then the one the document is routed to (by it's id)
 * even in a single-shard collection, with multiple replicas, documents with identical signatureField values will *only* be deleted on the 'leader' and not on any other replicas, because the leader does not propogate the {{AddUpdateCommand.updateTerm}} computed by the SignatureUpdateProcessorFactory to each of it's shards

----
In general, I don't think it's a good idea to try and "fix" the way SignatureUpdateProcessorFactory implements "De-Duplicaction" over fields that aren't the unique key field in SolrCloud. The original implementation of {{overwriteDupes=true}} leveraged low-level lucene functionality designed for "replacing" documents by their uniqueKey, but swapped in the "signatureField" instead of the "id" field and then also did a (local) Delete-By-Query looking for any docs with the same "id" field value.

This is already less efficient (locally) then a simple "replace" of a document based on it's uniqueKey – but if we also broadcast that DBQ to every shard in a SolrCloud usecase, it would be prohibitively slow, particularly since DBQs in Solr require a distrbuted "lock" preventing concurrent indexing, to insure the delete is done atomicly and shards are kept consistent (something that I didn't realize 9 years ago)

I've opened SOLR-15290 to track some new ideas for approaching this problem – starting with better docs, and better warnings/errors to prevent people from getting into problematic situations – and maybe some ideas for addressing "cloud level deduplication" in a scalable way.

> Distributed deduplication broken
> --------------------------------
>
>                 Key: SOLR-3473
>                 URL: https://issues.apache.org/jira/browse/SOLR-3473
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud, update
>    Affects Versions: 4.0-ALPHA
>            Reporter: Markus Jelsma
>            Priority: Major
>             Fix For: 4.9, 6.0
>
>         Attachments: SOLR-3473-trunk-2.patch, SOLR-3473.patch, SOLR-3473.patch
>
>
> Solr's deduplication via the SignatureUpdateProcessor is broken for distributed updates on SolrCloud.
> Mark Miller:
> {quote}
> Looking again at the SignatureUpdateProcessor code, I think that indeed this won't currently work with distrib updates. Could you file a JIRA issue for that? The problem is that we convert update commands into solr documents - and that can cause a loss of info if an update proc modifies the update command.
> I think the reason that you see a multiple values error when you try the other order is because of the lack of a document clone (the other issue I mentioned a few emails back). Addressing that won't solve your issue though - we have to come up with a way to propagate the currently lost info on the update command.
> {quote}
> Please see the ML thread for the full discussion: http://lucene.472066.n3.nabble.com/SolrCloud-deduplication-td3984657.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org