You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Eric Katherman <er...@knackhq.com> on 2014/09/27 17:12:45 UTC
Expunging Deletes
I'm running into memory issues and wondering if I should be using
expungeDeletes on commits. The server in question at the moment has
450k documents in the collection and represents 15GB on disk. There are
also 700k+ "Deleted Docs" and I'm guessing that is part of the disk
space consumption but I am not having any luck getting that cleared
out. I noticed the expungeDeletes=false in some of the log output
related to commit but didn't try setting it to true yet. Will this clear
those deleted documents and recover that space? Or should something
else already be managing that but maybe isn't configured correctly?
Our data is user specific data, each customer has their own database
structure so it varies with each user. They also add/remove data fairly
frequently in many cases. To compare another collection of the same
data type, there are 1M documents and about 120k deleted docs but disk
space is only 6.3GB.
Hoping someone can share some advice about how to manage this.
Thanks,
Eric
Re: Expunging Deletes
Posted by Upayavira <uv...@odoko.co.uk>.
Or consider separating frequently changing data into a different core
from the slow moving data, if you can, reducing the amount of data being
pushed around.
Upayavira
On Mon, Sep 29, 2014, at 09:16 PM, Bryan Bende wrote:
> You can try lowering the mergeFactor in solrconfig.xml to cause more
> merges
> to happen during normal indexing, which should result in more deleted
> documents being removed from the index, but there is a trade-off....
>
> http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor
>
> On Mon, Sep 29, 2014 at 2:14 PM, Eric Katherman <er...@knackhq.com> wrote:
>
> > Thanks for replying! Is there anything I could be doing to help prevent
> > the 14GB collection with 700k deleted docs before it tries removing them
> > and at that point running out of memory? Maybe just scheduled off-peak
> > optimize calls with expungeDeletes? Or is there some other config option I
> > could be using to help manage that a little better?
> >
> > Thanks!
> > Eric
> >
> >
> > On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com>
> > wrote:
> >
> > > Yes, expungeDeletes=true will remove all deleted docs from the disk but
> > it
> > > also requires merging all segments that have any deleted docs which, in
> > > your case, could mean a re-write of the entire index. So it'd be an
> > > expensive operation. Usually deletes are removed in the normal course of
> > > indexing as segments are merged together.
> > >
> > > On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com>
> > wrote:
> > >
> > >> I'm running into memory issues and wondering if I should be using
> > >> expungeDeletes on commits. The server in question at the moment has
> > 450k
> > >> documents in the collection and represents 15GB on disk. There are also
> > >> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> > >> consumption but I am not having any luck getting that cleared out. I
> > >> noticed the expungeDeletes=false in some of the log output related to
> > >> commit but didn't try setting it to true yet. Will this clear those
> > deleted
> > >> documents and recover that space? Or should something else already be
> > >> managing that but maybe isn't configured correctly?
> > >>
> > >> Our data is user specific data, each customer has their own database
> > >> structure so it varies with each user. They also add/remove data fairly
> > >> frequently in many cases. To compare another collection of the same
> > data
> > >> type, there are 1M documents and about 120k deleted docs but disk space
> > is
> > >> only 6.3GB.
> > >>
> > >> Hoping someone can share some advice about how to manage this.
> > >>
> > >> Thanks,
> > >> Eric
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> >
> >
Re: Expunging Deletes
Posted by Bryan Bende <bb...@gmail.com>.
You can try lowering the mergeFactor in solrconfig.xml to cause more merges
to happen during normal indexing, which should result in more deleted
documents being removed from the index, but there is a trade-off....
http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor
On Mon, Sep 29, 2014 at 2:14 PM, Eric Katherman <er...@knackhq.com> wrote:
> Thanks for replying! Is there anything I could be doing to help prevent
> the 14GB collection with 700k deleted docs before it tries removing them
> and at that point running out of memory? Maybe just scheduled off-peak
> optimize calls with expungeDeletes? Or is there some other config option I
> could be using to help manage that a little better?
>
> Thanks!
> Eric
>
>
> On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com>
> wrote:
>
> > Yes, expungeDeletes=true will remove all deleted docs from the disk but
> it
> > also requires merging all segments that have any deleted docs which, in
> > your case, could mean a re-write of the entire index. So it'd be an
> > expensive operation. Usually deletes are removed in the normal course of
> > indexing as segments are merged together.
> >
> > On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com>
> wrote:
> >
> >> I'm running into memory issues and wondering if I should be using
> >> expungeDeletes on commits. The server in question at the moment has
> 450k
> >> documents in the collection and represents 15GB on disk. There are also
> >> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> >> consumption but I am not having any luck getting that cleared out. I
> >> noticed the expungeDeletes=false in some of the log output related to
> >> commit but didn't try setting it to true yet. Will this clear those
> deleted
> >> documents and recover that space? Or should something else already be
> >> managing that but maybe isn't configured correctly?
> >>
> >> Our data is user specific data, each customer has their own database
> >> structure so it varies with each user. They also add/remove data fairly
> >> frequently in many cases. To compare another collection of the same
> data
> >> type, there are 1M documents and about 120k deleted docs but disk space
> is
> >> only 6.3GB.
> >>
> >> Hoping someone can share some advice about how to manage this.
> >>
> >> Thanks,
> >> Eric
> >>
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
>
>
Re: Expunging Deletes
Posted by Eric Katherman <er...@knackhq.com>.
Thanks for replying! Is there anything I could be doing to help prevent the 14GB collection with 700k deleted docs before it tries removing them and at that point running out of memory? Maybe just scheduled off-peak optimize calls with expungeDeletes? Or is there some other config option I could be using to help manage that a little better?
Thanks!
Eric
On Sep 29, 2014, at 9:35 AM, Shalin Shekhar Mangar <sh...@gmail.com> wrote:
> Yes, expungeDeletes=true will remove all deleted docs from the disk but it
> also requires merging all segments that have any deleted docs which, in
> your case, could mean a re-write of the entire index. So it'd be an
> expensive operation. Usually deletes are removed in the normal course of
> indexing as segments are merged together.
>
> On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com> wrote:
>
>> I'm running into memory issues and wondering if I should be using
>> expungeDeletes on commits. The server in question at the moment has 450k
>> documents in the collection and represents 15GB on disk. There are also
>> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
>> consumption but I am not having any luck getting that cleared out. I
>> noticed the expungeDeletes=false in some of the log output related to
>> commit but didn't try setting it to true yet. Will this clear those deleted
>> documents and recover that space? Or should something else already be
>> managing that but maybe isn't configured correctly?
>>
>> Our data is user specific data, each customer has their own database
>> structure so it varies with each user. They also add/remove data fairly
>> frequently in many cases. To compare another collection of the same data
>> type, there are 1M documents and about 120k deleted docs but disk space is
>> only 6.3GB.
>>
>> Hoping someone can share some advice about how to manage this.
>>
>> Thanks,
>> Eric
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
Re: Expunging Deletes
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yes, expungeDeletes=true will remove all deleted docs from the disk but it
also requires merging all segments that have any deleted docs which, in
your case, could mean a re-write of the entire index. So it'd be an
expensive operation. Usually deletes are removed in the normal course of
indexing as segments are merged together.
On Sat, Sep 27, 2014 at 8:42 PM, Eric Katherman <er...@knackhq.com> wrote:
> I'm running into memory issues and wondering if I should be using
> expungeDeletes on commits. The server in question at the moment has 450k
> documents in the collection and represents 15GB on disk. There are also
> 700k+ "Deleted Docs" and I'm guessing that is part of the disk space
> consumption but I am not having any luck getting that cleared out. I
> noticed the expungeDeletes=false in some of the log output related to
> commit but didn't try setting it to true yet. Will this clear those deleted
> documents and recover that space? Or should something else already be
> managing that but maybe isn't configured correctly?
>
> Our data is user specific data, each customer has their own database
> structure so it varies with each user. They also add/remove data fairly
> frequently in many cases. To compare another collection of the same data
> type, there are 1M documents and about 120k deleted docs but disk space is
> only 6.3GB.
>
> Hoping someone can share some advice about how to manage this.
>
> Thanks,
> Eric
>
--
Regards,
Shalin Shekhar Mangar.