You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Eric Katherman <er...@knackhq.com> on 2014/09/27 17:12:45 UTC

Expunging Deletes

I'm running into memory issues and wondering if I should be using 
expungeDeletes on commits.  The server in question at the moment has 
450k documents in the collection and represents 15GB on disk.  There are 
also 700k+ "Deleted Docs" and I'm guessing that is part of the disk 
space consumption but I am not having any luck getting that cleared 
out.  I noticed the expungeDeletes=false in some of the log output 
related to commit but didn't try setting it to true yet. Will this clear 
those deleted documents and recover that space?  Or should something 
else already be managing that but maybe isn't configured correctly?

Our data is user specific data, each customer has their own database 
structure so it varies with each user.  They also add/remove data fairly 
frequently in many cases.  To compare another collection of the same 
data type, there are 1M documents and about 120k deleted docs but disk 
space is only 6.3GB.

Hoping someone can share some advice about how to manage this.

Thanks,
Eric

Re: Expunging Deletes

Posted by Upayavira <uv...@odoko.co.uk>.
Or consider separating frequently changing data into a different core
from the slow moving data, if you can, reducing the amount of data being
pushed around.

Upayavira

On Mon, Sep 29, 2014, at 09:16 PM, Bryan Bende wrote:
> You can try lowering the mergeFactor in solrconfig.xml to cause more
> merges
> to happen during normal indexing, which should result in more deleted
> documents being removed from the index, but there is a trade-off....
> 
> http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor
> 
> On Mon, Sep 29, 2014 at 2:14 PM, Eric Katherman <er...@knackhq.com> wrote:
> 
> > Thanks for replying!  Is there anything I could be doing to help prevent
> > the 14GB collection with 700k deleted docs before it tries removing them
> > and at that point running out of memory?  Maybe just scheduled off-peak
> > optimize calls with expungeDeletes?  Or is there some other config option I
> > could be using to help manage that a little better?
> >
> > Thanks!
> > Eric
> >
> >
> > On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com>
> > wrote:
> >
> > > Yes, expungeDeletes=true will remove all deleted docs from the disk but
> > it
> > > also requires merging all segments that have any deleted docs which, in
> > > your case, could mean a re-write of the entire index. So it'd be an
> > > expensive operation. Usually deletes are removed in the normal course of
> > > indexing as segments are merged together.
> > >
> > > On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com>
> > wrote:
> > >
> > >> I'm running into memory issues and wondering if I should be using
> > >> expungeDeletes on commits.  The server in question at the moment has
> > 450k
> > >> documents in the collection and represents 15GB on disk.  There are also
> > >> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> > >> consumption but I am not having any luck getting that cleared out.  I
> > >> noticed the expungeDeletes=false in some of the log output related to
> > >> commit but didn't try setting it to true yet. Will this clear those
> > deleted
> > >> documents and recover that space?  Or should something else already be
> > >> managing that but maybe isn't configured correctly?
> > >>
> > >> Our data is user specific data, each customer has their own database
> > >> structure so it varies with each user.  They also add/remove data fairly
> > >> frequently in many cases.  To compare another collection of the same
> > data
> > >> type, there are 1M documents and about 120k deleted docs but disk space
> > is
> > >> only 6.3GB.
> > >>
> > >> Hoping someone can share some advice about how to manage this.
> > >>
> > >> Thanks,
> > >> Eric
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> >
> >

Re: Expunging Deletes

Posted by Bryan Bende <bb...@gmail.com>.
You can try lowering the mergeFactor in solrconfig.xml to cause more merges
to happen during normal indexing, which should result in more deleted
documents being removed from the index, but there is a trade-off....

http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor

On Mon, Sep 29, 2014 at 2:14 PM, Eric Katherman <er...@knackhq.com> wrote:

> Thanks for replying!  Is there anything I could be doing to help prevent
> the 14GB collection with 700k deleted docs before it tries removing them
> and at that point running out of memory?  Maybe just scheduled off-peak
> optimize calls with expungeDeletes?  Or is there some other config option I
> could be using to help manage that a little better?
>
> Thanks!
> Eric
>
>
> On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com>
> wrote:
>
> > Yes, expungeDeletes=true will remove all deleted docs from the disk but
> it
> > also requires merging all segments that have any deleted docs which, in
> > your case, could mean a re-write of the entire index. So it'd be an
> > expensive operation. Usually deletes are removed in the normal course of
> > indexing as segments are merged together.
> >
> > On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com>
> wrote:
> >
> >> I'm running into memory issues and wondering if I should be using
> >> expungeDeletes on commits.  The server in question at the moment has
> 450k
> >> documents in the collection and represents 15GB on disk.  There are also
> >> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> >> consumption but I am not having any luck getting that cleared out.  I
> >> noticed the expungeDeletes=false in some of the log output related to
> >> commit but didn't try setting it to true yet. Will this clear those
> deleted
> >> documents and recover that space?  Or should something else already be
> >> managing that but maybe isn't configured correctly?
> >>
> >> Our data is user specific data, each customer has their own database
> >> structure so it varies with each user.  They also add/remove data fairly
> >> frequently in many cases.  To compare another collection of the same
> data
> >> type, there are 1M documents and about 120k deleted docs but disk space
> is
> >> only 6.3GB.
> >>
> >> Hoping someone can share some advice about how to manage this.
> >>
> >> Thanks,
> >> Eric
> >>
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
>
>

Re: Expunging Deletes

Posted by Eric Katherman <er...@knackhq.com>.
Thanks for replying!  Is there anything I could be doing to help prevent the 14GB collection with 700k deleted docs before it tries removing them and at that point running out of memory?  Maybe just scheduled off-peak optimize calls with expungeDeletes?  Or is there some other config option I could be using to help manage that a little better?

Thanks!
Eric


On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com> wrote:

> Yes, expungeDeletes=true will remove all deleted docs from the disk but it
> also requires merging all segments that have any deleted docs which, in
> your case, could mean a re-write of the entire index. So it'd be an
> expensive operation. Usually deletes are removed in the normal course of
> indexing as segments are merged together.
> 
> On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com> wrote:
> 
>> I'm running into memory issues and wondering if I should be using
>> expungeDeletes on commits.  The server in question at the moment has 450k
>> documents in the collection and represents 15GB on disk.  There are also
>> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
>> consumption but I am not having any luck getting that cleared out.  I
>> noticed the expungeDeletes=false in some of the log output related to
>> commit but didn't try setting it to true yet. Will this clear those deleted
>> documents and recover that space?  Or should something else already be
>> managing that but maybe isn't configured correctly?
>> 
>> Our data is user specific data, each customer has their own database
>> structure so it varies with each user.  They also add/remove data fairly
>> frequently in many cases.  To compare another collection of the same data
>> type, there are 1M documents and about 120k deleted docs but disk space is
>> only 6.3GB.
>> 
>> Hoping someone can share some advice about how to manage this.
>> 
>> Thanks,
>> Eric
>> 
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.


Re: Expunging Deletes

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yes, expungeDeletes=true will remove all deleted docs from the disk but it
also requires merging all segments that have any deleted docs which, in
your case, could mean a re-write of the entire index. So it'd be an
expensive operation. Usually deletes are removed in the normal course of
indexing as segments are merged together.

On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com> wrote:

> I'm running into memory issues and wondering if I should be using
> expungeDeletes on commits.  The server in question at the moment has 450k
> documents in the collection and represents 15GB on disk.  There are also
> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> consumption but I am not having any luck getting that cleared out.  I
> noticed the expungeDeletes=false in some of the log output related to
> commit but didn't try setting it to true yet. Will this clear those deleted
> documents and recover that space?  Or should something else already be
> managing that but maybe isn't configured correctly?
>
> Our data is user specific data, each customer has their own database
> structure so it varies with each user.  They also add/remove data fairly
> frequently in many cases.  To compare another collection of the same data
> type, there are 1M documents and about 120k deleted docs but disk space is
> only 6.3GB.
>
> Hoping someone can share some advice about how to manage this.
>
> Thanks,
> Eric
>



-- 
Regards,
Shalin Shekhar Mangar.