Can index size increase when no updates/optimizes are happening?

[Yury Kats <yu...@yahoo.com>]

One of my users observed that the index size (in bytes)
increased over night. There was no indexing activity
at that time, only querying was taking place.

Running "optimize" brought the index size back down to
what it was when indexing finished the day before.

What could explain that?

Re: Can index size increase when no updates/optimizes are happening?

[eanand333 <ea...@gmail.com>]

This is what we do,
A user logs in - enter s a few documents in a particular domain, say A, B
or C - logs out.
Say B is the most commonly used domain. The increase in index size is
drastic only in this particular domain.

So unless a user logs in there s no question of documents being submitted
or any indexing activity.
>From the logs i see no user logged in during that time frame.

Adding to this we have to take a back up of the index every day and that s
how we even came to know that such a problem even existed.

Is there a possibility that i could schedule an optimize to run at a
specific time during the day and hence try control the index s file size?






On Sat, Mar 16, 2013 at 6:51 PM, Erick Erickson [via Lucene] <
ml-node+s472066n4047962h77@n3.nabble.com> wrote:

> Well, if nothing is going on at all, it's hard to see why the index would
> increase. So I suspect _something_ is going on. Possibilities:
>
> 1> you have indexing activity going on. Even if it's just replacing docs
> that already exist, which is actually an add and a delete the index will
> grow for a while; the deleted info isn't removed from the index until the
> segment is merged, which happens unpredictably (well, actually predictably
> but not on a schedule you enforce). So the index would
> grow/shrink/grow/shrink. Do you have automatic process in the background
> that push docs to the index?
>
> 2> You forceMerge (optimize), in which case the index will at least double
> in size temporarily.
>
> 3> You are replicating. While the replication goes on, especially if your
> index has changed greatly, then your index could double.
>
> None of these fit the symptoms you describe very well mind you. It's
> suspicious that these increases last long enough for you to see them
> predictably in the morning, unless, say, a background process indexes
> things regularly and you always look at the same time... unlikely at best.
>
> But the fact that your index goes from 18G to 8G strongly suggests that
> you're doing a forceMerge/optimize when you see it bump up to 18G.
>
> Best
> Erick
>
>
>
>
> On Sat, Mar 16, 2013 at 4:52 AM, eanand333 <[hidden email]<http://user/SendEmail.jtp?type=node&node=4047962&i=0>>
> wrote:
>
> > Hi, I am kind of new in here. Got the same question...
> > I am using Java version 1.6 and Lucene version 3.3.
> > Can the index file size increase automatically over night?
> > During the evening i see the size around 11GB, next day morning i see it
> to
> > be 18GB and again the size reduces around 8GB.
> > I have checked the logs and i am sure that there was no user activity
> > during
> > this period.
> >
> > Mr. Eric has suggested this to be a user error, if that s true, i would
> > like
> > to know what are the possible errors which could result in the rapid
> > increase in index file size ?
> >
> > Or what are the other possibilities for the index size to increase
> > exponentially?
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047962.html
>  To unsubscribe from Can index size increase when no updates/optimizes are
> happening?, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3334022&code=ZWFuYW5kLjA2OUBnbWFpbC5jb218MzMzNDAyMnwtNTIyMjg4NDE1>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4048444.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Can index size increase when no updates/optimizes are happening?

[Erick Erickson <er...@gmail.com>]

Well, if nothing is going on at all, it's hard to see why the index would
increase. So I suspect _something_ is going on. Possibilities:

1> you have indexing activity going on. Even if it's just replacing docs
that already exist, which is actually an add and a delete the index will
grow for a while; the deleted info isn't removed from the index until the
segment is merged, which happens unpredictably (well, actually predictably
but not on a schedule you enforce). So the index would
grow/shrink/grow/shrink. Do you have automatic process in the background
that push docs to the index?

2> You forceMerge (optimize), in which case the index will at least double
in size temporarily.

3> You are replicating. While the replication goes on, especially if your
index has changed greatly, then your index could double.

None of these fit the symptoms you describe very well mind you. It's
suspicious that these increases last long enough for you to see them
predictably in the morning, unless, say, a background process indexes
things regularly and you always look at the same time... unlikely at best.

But the fact that your index goes from 18G to 8G strongly suggests that
you're doing a forceMerge/optimize when you see it bump up to 18G.

Best
Erick




On Sat, Mar 16, 2013 at 4:52 AM, eanand333 <ea...@gmail.com> wrote:

> Hi, I am kind of new in here. Got the same question...
> I am using Java version 1.6 and Lucene version 3.3.
> Can the index file size increase automatically over night?
> During the evening i see the size around 11GB, next day morning i see it to
> be 18GB and again the size reduces around 8GB.
> I have checked the logs and i am sure that there was no user activity
> during
> this period.
>
> Mr. Eric has suggested this to be a user error, if that s true, i would
> like
> to know what are the possible errors which could result in the rapid
> increase in index file size ?
>
> Or what are the other possibilities for the index size to increase
> exponentially?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Can index size increase when no updates/optimizes are happening?

[eanand333 <ea...@gmail.com>]

Hi, I am kind of new in here. Got the same question...
I am using Java version 1.6 and Lucene version 3.3.
Can the index file size increase automatically over night?
During the evening i see the size around 11GB, next day morning i see it to
be 18GB and again the size reduces around 8GB.
I have checked the logs and i am sure that there was no user activity during
this period.

Mr. Eric has suggested this to be a user error, if that s true, i would like
to know what are the possible errors which could result in the rapid
increase in index file size ?

Or what are the other possibilities for the index size to increase
exponentially?

Thanks



--
View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Can index size increase when no updates/optimizes are happening?

[Erick Erickson <er...@gmail.com>]

I'd guess user error in the absence of stronger information. No
docs added, no optimized kicked off, no replication and I
can't think of any reason the index size should increase.

Erick

2011/9/15 Yury Kats <yu...@yahoo.com>:
> On 9/14/2011 2:36 PM, Erick Erickson wrote:
>> What is the machine used for? Was your user looking at
>> a master? Slave? Something used for both?
>
> Stand-alone machine with multiple Solr cores. No replication.
>
>> Measuring the size of all the files in the index? Or looking
>> at memory?
>
> Disk space.
>
>> The index files shouldn't be getting bigger unless there
>> were indexing operations going on.
>
> That's what I thought.
>
>> Is it at all possible that
>> DIH was configured to run automatically (or any other
>> indexing job for that matter) and your user didn't realize it?
>
> There's no DIH, but there is a custom app that submit docs
> for indexing via SolrJ. Supposedly, Solr logs were not showing
> any updates over night, so the assumption was that no new docs
> were added.
>
> I'd write it off as a user error, but wanted to double check with
> the community that no other internal Solr/Lucene task can change the index
> file size in the absence of submits.
>

Re: Can index size increase when no updates/optimizes are happening?

[Yury Kats <yu...@yahoo.com>]

On 9/14/2011 2:36 PM, Erick Erickson wrote:
> What is the machine used for? Was your user looking at
> a master? Slave? Something used for both?

Stand-alone machine with multiple Solr cores. No replication.

> Measuring the size of all the files in the index? Or looking
> at memory?

Disk space.

> The index files shouldn't be getting bigger unless there
> were indexing operations going on. 

That's what I thought.

> Is it at all possible that
> DIH was configured to run automatically (or any other
> indexing job for that matter) and your user didn't realize it?

There's no DIH, but there is a custom app that submit docs
for indexing via SolrJ. Supposedly, Solr logs were not showing
any updates over night, so the assumption was that no new docs
were added.

I'd write it off as a user error, but wanted to double check with
the community that no other internal Solr/Lucene task can change the index
file size in the absence of submits.

Re: Can index size increase when no updates/optimizes are happening?

[Erick Erickson <er...@gmail.com>]

What is the machine used for? Was your user looking at
a master? Slave? Something used for both?

Measuring the size of all the files in the index? Or looking
at memory?

The index files shouldn't be getting bigger unless there
were indexing operations going on. Is it at all possible that
DIH was configured to run automatically (or any other
indexing job for that matter) and your user didn't realize it?

Best
Erick

2011/9/13 Yury Kats <yu...@yahoo.com>:
> One of my users observed that the index size (in bytes)
> increased over night. There was no indexing activity
> at that time, only querying was taking place.
>
> Running "optimize" brought the index size back down to
> what it was when indexing finished the day before.
>
> What could explain that?
>
>