Deleted Docs increasing in Solr 6.1.0

[vishal patel <vi...@outlook.com>]

We have 2 shards and 2 replicas in a testing environment.Deleted Docs are 18749 for one collection[documents].I have attached a screenshot of solr admin panel.
(1) Would there any impact on disk size if deleted docs will increase?
(2) We try to remove deleted doc by executing command : curl "http://50.40.30.20:8983/solr/documents/update" -H "Content-Type:text/xml" --data-binary "<commit expungeDeletes='true'/>"
After executing this command, we checked in solr admin but no changed in deleted docs. How can we remove the deleted docs?

Note: default merge policy[TieredMergePolicy] used.
<http://aka.ms/weboutlook>


Regards,
Vishal

Re: Deleted Docs increasing in Solr 6.1.0

[Erick Erickson <er...@gmail.com>]

First of all I wouldn’t worry about it unless you have a _significant_ number of deleted docs. The default TMP as of around Solr 7.5 should accumulate up to around 33% deleted docs. Prior to 7.5, the number of deleted docs could hover around 50% depending on the access pattern.

expungeDeletes will not merge any segment with < 10% deleted docs as per the documentation.

NOTE: the numbers of deleted docs I mentioned above is wildly variable on small collections. For instance, an index with 100K docs could _easily_ have a larger percentage of deleted docs than above since merge only happens when there are 10 (default) like-sized segments to merge.

Bottom-line: 
- Yes, deleted docs occupy space on the disk.
- expungeDeletes only works on segments with > 10% deleted docs
- 90% of the time, worrying about the number of deleted docs is a waste of time. Lucene is quite efficient at skipping working with deleted docs so any performance gains, especially on small collections, is hard to measure.

Best,
Erick

> On Nov 4, 2019, at 5:01 AM, vishal patel <vi...@outlook.com> wrote:
> 
> We have 2 shards and 2 replicas in a testing environment.Deleted Docs are 18749 for one collection[documents].I have attached a screenshot of solr admin panel.
> (1) Would there any impact on disk size if deleted docs will increase?
> (2) We try to remove deleted doc by executing command : curl "http://50.40.30.20:8983/solr/documents/update" -H "Content-Type:text/xml" --data-binary "<commit expungeDeletes='true'/>"
> After executing this command, we checked in solr admin but no changed in deleted docs. How can we remove the deleted docs?
> 
> Note: default merge policy[TieredMergePolicy] used.
> 
> 
> Regards,
> Vishal