Re: Rejecting document already existing in different shard.

[Dmitry Kan <so...@gmail.com>]

Hi,

Although we use logical sharding, there are cases in our environment as you
described. We handle them manually:

0. prepare new version of a document
1. remove the old version of the document
2. post it and commit

With logical sharding it is relatively easy, but we do need to store
location metadata in a DB.

In your case, have you had a look onto this:

http://wiki.apache.org/solr/Deduplication

Other things that come to mind: store the parameters of hashing and then
find a link between new and parameters of the "same" document.

Dmitry


On Wed, Mar 13, 2013 at 11:34 PM, Marcin Rzewucki <mr...@gmail.com>wrote:

> Hi there,
>
> Let's say we use custom hashing algorithm and there is a document already
> indexed in "shard1". After some time the same document has changed and
> should be indexed to "shard2" (because of routing rules used in indexing
> program). It has been indexed without issues and as a result 2 "almost" the
> same documents are in different shards. In my case, they are duplicates for
> the end user. Is it possible to reject a document if it already exists in
> different shard ? It would be even easier to handle such cases prior to
> adding new with the same ID.
>
> Regards.
>