indexing while optimizing

[Laimonas Simutis <la...@gmail.com>]

Hey,

my SOLR setup looks like the following:

server running apache-tomcat with solr1.2, index size is about 1G (a
bit more than 4 million documents).

I have another machine that basically every minute or so sends some
documents to be indexed. I have autocommit turned on with maxDocs:
5000, maxTime: 300000ms.

Also on the server cron job runs twice per day to optimize the index.
And sometimes it happens that the index messages come at the time of
optimize running. I know that running optimize on an index that is
about to have documents added or deleted is not that useful, but it
did help eliminate "too many file handles open" problem.

Is it bad that I try to index when the optimize is running? I do see
failures on the client side from time to time, but the messages get
resent and indexed eventually.

One recurring problem is that once per 36 hours or so SOLR server
becomes really unresponsive, just spinning crazy on CPU and it is all
in java (solr) process. When I try to shut down apache, apache goes
down but the java process is left running. I am trying to pin point
where the problem is, and wonder if my indexing-commit is not right.
The box is solely dedicated for solr, so there is really nothing else
running on it.

Any pointers or observations appreciated.

thanks,

L

Re: indexing while optimizing

[Laimonas Simutis <la...@gmail.com>]

Upgraded to solr 1.3, and also moved it to a different machine (more
CPU the rest is the same). All the problems seem to have gone away...
Should have upgraded from the start.

On Fri, Feb 27, 2009 at 2:50 PM, Laimonas Simutis <la...@gmail.com> wrote:
> Hey,
>
> my SOLR setup looks like the following:
>
> server running apache-tomcat with solr1.2, index size is about 1G (a
> bit more than 4 million documents).
>
> I have another machine that basically every minute or so sends some
> documents to be indexed. I have autocommit turned on with maxDocs:
> 5000, maxTime: 300000ms.
>
> Also on the server cron job runs twice per day to optimize the index.
> And sometimes it happens that the index messages come at the time of
> optimize running. I know that running optimize on an index that is
> about to have documents added or deleted is not that useful, but it
> did help eliminate "too many file handles open" problem.
>
> Is it bad that I try to index when the optimize is running? I do see
> failures on the client side from time to time, but the messages get
> resent and indexed eventually.
>
> One recurring problem is that once per 36 hours or so SOLR server
> becomes really unresponsive, just spinning crazy on CPU and it is all
> in java (solr) process. When I try to shut down apache, apache goes
> down but the java process is left running. I am trying to pin point
> where the problem is, and wonder if my indexing-commit is not right.
> The box is solely dedicated for solr, so there is really nothing else
> running on it.
>
> Any pointers or observations appreciated.
>
> thanks,
>
> L
>