You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yang <te...@gmail.com> on 2012/07/11 05:04:05 UTC
index searcher leading to system freeze ?
I'm running 8 index searchers java processes on a 8-core node.
They all read from the same lucene index on local hard drive.
the index contains about 20million docs, each doc is a small record with
about 200 bytes.
total size is about 5GB.
it worked fine before, but recently (I don't know what changed) I found
that once the index searchers
are started, kswapd and kjournald take up 100% cpu (but this is not such a
bad thing, cuz the 8 cores
offer 800% cpu), and quickly the system comes to a freeze.
I do NOT have swap on the system, so I don't understand why kswapd and
kjournald should be so busy.
But even if these 2 kernel processes are busy, that's no reason for the
system to freeze either.
Does this sound like a kernel bug?
the system is a EC2 instance:
Linux blahblah.mycompany.com 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28
EST 2008 x86_64 x86_64 x86_64 GNU/Linux
Thanks a lot
yang
Re: index searcher leading to system freeze ?
Posted by Israel Tsadok <it...@gmail.com>.
I'm not sure this is at all related, but we've had high cpu loads on our
servers due to the leap second kernel bug -
http://serverfault.com/questions/403732/<http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second>
.
On Wed, Jul 11, 2012 at 4:00 PM, Simon Willnauer
<si...@gmail.com>wrote:
> are you closing your underlying IndexReaders properly?
>
> simon
>
> On Wed, Jul 11, 2012 at 5:04 AM, Yang <te...@gmail.com> wrote:
> > I'm running 8 index searchers java processes on a 8-core node.
> > They all read from the same lucene index on local hard drive.
> >
> >
> > the index contains about 20million docs, each doc is a small record with
> > about 200 bytes.
> > total size is about 5GB.
> >
> > it worked fine before, but recently (I don't know what changed) I found
> > that once the index searchers
> > are started, kswapd and kjournald take up 100% cpu (but this is not such
> a
> > bad thing, cuz the 8 cores
> > offer 800% cpu), and quickly the system comes to a freeze.
> >
> > I do NOT have swap on the system, so I don't understand why kswapd and
> > kjournald should be so busy.
> > But even if these 2 kernel processes are busy, that's no reason for the
> > system to freeze either.
> > Does this sound like a kernel bug?
> >
> >
> > the system is a EC2 instance:
> > Linux blahblah.mycompany.com 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15
> 12:34:28
> > EST 2008 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > Thanks a lot
> > yang
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: index searcher leading to system freeze ?
Posted by Simon Willnauer <si...@gmail.com>.
are you closing your underlying IndexReaders properly?
simon
On Wed, Jul 11, 2012 at 5:04 AM, Yang <te...@gmail.com> wrote:
> I'm running 8 index searchers java processes on a 8-core node.
> They all read from the same lucene index on local hard drive.
>
>
> the index contains about 20million docs, each doc is a small record with
> about 200 bytes.
> total size is about 5GB.
>
> it worked fine before, but recently (I don't know what changed) I found
> that once the index searchers
> are started, kswapd and kjournald take up 100% cpu (but this is not such a
> bad thing, cuz the 8 cores
> offer 800% cpu), and quickly the system comes to a freeze.
>
> I do NOT have swap on the system, so I don't understand why kswapd and
> kjournald should be so busy.
> But even if these 2 kernel processes are busy, that's no reason for the
> system to freeze either.
> Does this sound like a kernel bug?
>
>
> the system is a EC2 instance:
> Linux blahblah.mycompany.com 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28
> EST 2008 x86_64 x86_64 x86_64 GNU/Linux
>
>
> Thanks a lot
> yang
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org