You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Rasmussen, Chris" <cr...@mitre.org> on 2013/08/08 18:47:44 UTC

SolrCloud "deletedDocs"

I'm running a 4.2 SOLRCloud instance with multiple servers/shards.  As I'm indexing data, I review the results of the STATUS commands and note an extremely high number of "deletedDocs".  I've combed through the source data to verify whether I'm sending duplicate documents ids, but haven't been able to find any.  I'm starting to wonder whether the field is a red herring?

Is the deleted document counter an accurate reflection of documents marked deleted in the collection?  My assumption is that if I send a document with the same document id, Solr will marked the document as deleted and then insert the new one.  Then at merge time the deleted documents are purged from the index.  I've noted the the total deleted document count will go down with the indexes are merged.

Chris Rasmussen

Re: SolrCloud "deletedDocs"

Posted by Shawn Heisey <so...@elyograg.org>.
On 8/8/2013 10:47 AM, Rasmussen, Chris wrote:
> I'm running a 4.2 SOLRCloud instance with multiple servers/shards.  As I'm indexing data, I review the results of the STATUS commands and note an extremely high number of "deletedDocs".  I've combed through the source data to verify whether I'm sending duplicate documents ids, but haven't been able to find any.  I'm starting to wonder whether the field is a red herring?
>
> Is the deleted document counter an accurate reflection of documents marked deleted in the collection?  My assumption is that if I send a document with the same document id, Solr will marked the document as deleted and then insert the new one.  Then at merge time the deleted documents are purged from the index.  I've noted the the total deleted document count will go down with the indexes are merged.

The deletedDocs number is the accurate count of documents deleted in 
that specific SolrCore -- that particular replica of the shard.

You are correct about what happens if you index a document that already 
exists with the same value in the uniqueKey field.  The old one is 
deleted (which just marks it as deleted) and the new one is inserted. 
You are also correct about deleted documents not actually disappearing 
from an index segment until that segment is merged, or you do an optimize.

If you restart any of your Solr servers, or a problem happens such that 
the cluster timeout is exceeded and everything gets out of sync, then 
you might end up in a situation where the transaction log is replayed, 
which reindex old documents and might increase the deleted count.

Thanks,
Shawn