index growing with updates

["Nagelberg, Kallin" <KN...@globeandmail.com>]

Hey,

If I add a document to the index that already exists (same uniquekey) what is the expected behavior? I would imagine that if the document is the same then the index should not grow, but mine appears to be growing. Any ideas?

Thanks,
-Kallin Nagelberg

Re: index growing with updates

[Chris Rode <ci...@gmail.com>]

As I understand it one of the easiest ways of ensuring a purge is to trigger
an optimisation. This happens periodically depending on your configuration
or you can trigger it manually.

Regards

Chris

On 3 June 2010 18:36, Nagelberg, Kallin <KN...@globeandmail.com> wrote:

> Is there a way to trigger a purge, or under what conditions does it occur?
>
> -Kallin Nagelberg
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Thursday, June 03, 2010 12:40 PM
> To: solr-user@lucene.apache.org
> Subject: Re: index growing with updates
>
> Assuming your config is set up to replace unique keys, you're really
> doing a delete and an add (under the covers). It could very well be that
> the deleted version of the document is still in your index taking up
> space and will be until it is purged.
>
> HTH
> Erick
>
> On Thu, Jun 3, 2010 at 10:22 AM, Nagelberg, Kallin <
> KNagelberg@globeandmail.com> wrote:
>
> > Hey,
> >
> > If I add a document to the index that already exists (same uniquekey)
> what
> > is the expected behavior? I would imagine that if the document is the
> same
> > then the index should not grow, but mine appears to be growing. Any
> ideas?
> >
> > Thanks,
> > -Kallin Nagelberg
> >
> >
>

RE: index growing with updates

[Chris Hostetter <ho...@fucit.org>]

: Ok so I think that Solr (lucene) will only remove deleted/updated 
: documents from the disk after an optimize or after an 'expungeDeletes' 
: request. Is there a way to trigger the expunsion (new word) across the 
: entire index? I tried :

deletes are removed when segments are merged -- an optimize merges all 
segments, so it forcibley removes all deleted docs, but regular merges as 
documents are added/updated will clean things up periodicly -- so if you 
have a fixed set of documents that you keep updating over and over your 
index size will not grow with out bounds -- it will ossilate between a min 
(completely optimized) and a max (lots of segments with lots of deletions 
just about to be merged)



-Hoss

RE: index growing with updates

["Nagelberg, Kallin" <KN...@globeandmail.com>]

Ok so I think that Solr (lucene) will only remove deleted/updated documents from the disk after an optimize or after an 'expungeDeletes' request. Is there a way to trigger the expunsion (new word) across the entire index? I tried :

final UpdateRequest request = new UpdateRequest()
request.setParam("expungeDeletes","true");
request.add someofmydocs
server.sendrequest..


But that didn't seem to do the trick as I know I have about 7 Gigs of documents that should be removed from the disk and the index size hasn't really budged.

Any ideas?

Thanks,
Kallin Nagelberg





-----Original Message-----
From: Nagelberg, Kallin 
Sent: Thursday, June 03, 2010 1:36 PM
To: 'solr-user@lucene.apache.org'
Subject: RE: index growing with updates

Is there a way to trigger a purge, or under what conditions does it occur?

-Kallin Nagelberg

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Thursday, June 03, 2010 12:40 PM
To: solr-user@lucene.apache.org
Subject: Re: index growing with updates

Assuming your config is set up to replace unique keys, you're really
doing a delete and an add (under the covers). It could very well be that
the deleted version of the document is still in your index taking up
space and will be until it is purged.

HTH
Erick

On Thu, Jun 3, 2010 at 10:22 AM, Nagelberg, Kallin <
KNagelberg@globeandmail.com> wrote:

> Hey,
>
> If I add a document to the index that already exists (same uniquekey) what
> is the expected behavior? I would imagine that if the document is the same
> then the index should not grow, but mine appears to be growing. Any ideas?
>
> Thanks,
> -Kallin Nagelberg
>
>

RE: index growing with updates

["Nagelberg, Kallin" <KN...@globeandmail.com>]

Is there a way to trigger a purge, or under what conditions does it occur?

-Kallin Nagelberg

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Thursday, June 03, 2010 12:40 PM
To: solr-user@lucene.apache.org
Subject: Re: index growing with updates

Assuming your config is set up to replace unique keys, you're really
doing a delete and an add (under the covers). It could very well be that
the deleted version of the document is still in your index taking up
space and will be until it is purged.

HTH
Erick

On Thu, Jun 3, 2010 at 10:22 AM, Nagelberg, Kallin <
KNagelberg@globeandmail.com> wrote:

> Hey,
>
> If I add a document to the index that already exists (same uniquekey) what
> is the expected behavior? I would imagine that if the document is the same
> then the index should not grow, but mine appears to be growing. Any ideas?
>
> Thanks,
> -Kallin Nagelberg
>
>

Re: index growing with updates

[Erick Erickson <er...@gmail.com>]

Assuming your config is set up to replace unique keys, you're really
doing a delete and an add (under the covers). It could very well be that
the deleted version of the document is still in your index taking up
space and will be until it is purged.

HTH
Erick

On Thu, Jun 3, 2010 at 10:22 AM, Nagelberg, Kallin <
KNagelberg@globeandmail.com> wrote:

> Hey,
>
> If I add a document to the index that already exists (same uniquekey) what
> is the expected behavior? I would imagine that if the document is the same
> then the index should not grow, but mine appears to be growing. Any ideas?
>
> Thanks,
> -Kallin Nagelberg
>
>