You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@roller.apache.org by Rudman Max <mr...@steelbrick.com> on 2005/06/29 04:09:32 UTC
Orphaned file descriptors on /roller-index
We've been testing Roller with some pretty high load (about 500
concurrent users) running search (which reads from the index) and
posts (which writes to the index) transactions. After a while, we'd
run out of file handles which froze Tomcat because it could no longer
open sockets to accept incoming connections. Our sysadmin told me
there were a bunch of orphaned file descriptors on files in /roller-
index directory. I am not sure if that's the reason for Tomcat
process running out of files but it seems likely. Has anybody ever
experienced this problem with Roller search index?
Max
Re: Orphaned file descriptors on /roller-index
Posted by Allen Gilliland <Al...@Sun.COM>.
I haven't really looked too much at the search code or done much work
with Lucene, but I would agree that it doesn't seem to make much sense
to try and update the index immediately every time someone posts a new
entry. I think a scheduled task that simply updates the entire index
periodically, maybe every 30-60 minutes, would be fine.
-- Allen
Rudman Max wrote:
> When you say "internal search engine" do you mean Lucene? I presume
> by "external" you mean something like Google? At any rate, I think
> the problem is either in Lucene or in that "edu.oswego" library you
> are using to schedule index writes. I think what's happening is that
> at sustained high rate of posts the threads wanting to write to
> Lucene index are piling up and causing the problem. Sadly, I don't
> think concurrency level needed to expose this bug is very high at
> all. I've been able to reproduce this with as few as 5 concurrent
> users making posts.
>
> My fear is that this is something fundamental to the way Lucene
> manages its indexes and I suspect it's not an easy fix. I mean this
> is the kind of problem that database vendors have to deal with pretty
> sophisticated solutions. How receptive would you (and the community)
> be to a proposal for changing real-time writes to some sort of
> batched mode where a process is run periodically which is responsible
> for indexing un-indexed entries?
>
> Max
>
>
> On Jun 28, 2005, at 10:52 PM, Allen Gilliland wrote:
>
>> I'm not sure if we've ever isolated things down to exactly that
>> problem, but for blogs.sun.com we've definitely had a number of
>> problems with the built in search engine. I believe a number of the
>> problems have been fixed, so maybe if you aren't using the latest
>> cvs already then you can run your tests against the 1.2 release
>> coming up and see what happens. Unfortunately we won't be the best
>> help with search problems because we use an external search engine
>> and so we currently have Roller's built in search disabled.
>>
>> -- Allen
>>
>>
>> Rudman Max wrote:
>>
>>
>>> We've been testing Roller with some pretty high load (about 500
>>> concurrent users) running search (which reads from the index) and
>>> posts (which writes to the index) transactions. After a while,
>>> we'd run out of file handles which froze Tomcat because it could
>>> no longer open sockets to accept incoming connections. Our
>>> sysadmin told me there were a bunch of orphaned file descriptors
>>> on files in /roller- index directory. I am not sure if that's the
>>> reason for Tomcat process running out of files but it seems
>>> likely. Has anybody ever experienced this problem with Roller
>>> search index?
>>>
>>> Max
>>>
>>
>
Re: Orphaned file descriptors on /roller-index
Posted by Rudman Max <mr...@steelbrick.com>.
When you say "internal search engine" do you mean Lucene? I presume
by "external" you mean something like Google? At any rate, I think
the problem is either in Lucene or in that "edu.oswego" library you
are using to schedule index writes. I think what's happening is that
at sustained high rate of posts the threads wanting to write to
Lucene index are piling up and causing the problem. Sadly, I don't
think concurrency level needed to expose this bug is very high at
all. I've been able to reproduce this with as few as 5 concurrent
users making posts.
My fear is that this is something fundamental to the way Lucene
manages its indexes and I suspect it's not an easy fix. I mean this
is the kind of problem that database vendors have to deal with pretty
sophisticated solutions. How receptive would you (and the community)
be to a proposal for changing real-time writes to some sort of
batched mode where a process is run periodically which is responsible
for indexing un-indexed entries?
Max
On Jun 28, 2005, at 10:52 PM, Allen Gilliland wrote:
> I'm not sure if we've ever isolated things down to exactly that
> problem, but for blogs.sun.com we've definitely had a number of
> problems with the built in search engine. I believe a number of
> the problems have been fixed, so maybe if you aren't using the
> latest cvs already then you can run your tests against the 1.2
> release coming up and see what happens. Unfortunately we won't be
> the best help with search problems because we use an external
> search engine and so we currently have Roller's built in search
> disabled.
>
> -- Allen
>
>
> Rudman Max wrote:
>
>
>> We've been testing Roller with some pretty high load (about 500
>> concurrent users) running search (which reads from the index) and
>> posts (which writes to the index) transactions. After a while,
>> we'd run out of file handles which froze Tomcat because it could
>> no longer open sockets to accept incoming connections. Our
>> sysadmin told me there were a bunch of orphaned file descriptors
>> on files in /roller- index directory. I am not sure if that's the
>> reason for Tomcat process running out of files but it seems
>> likely. Has anybody ever experienced this problem with Roller
>> search index?
>>
>> Max
>>
>
Re: Orphaned file descriptors on /roller-index
Posted by Allen Gilliland <Al...@Sun.COM>.
I'm not sure if we've ever isolated things down to exactly that problem,
but for blogs.sun.com we've definitely had a number of problems with the
built in search engine. I believe a number of the problems have been
fixed, so maybe if you aren't using the latest cvs already then you can
run your tests against the 1.2 release coming up and see what happens.
Unfortunately we won't be the best help with search problems because we
use an external search engine and so we currently have Roller's built in
search disabled.
-- Allen
Rudman Max wrote:
> We've been testing Roller with some pretty high load (about 500
> concurrent users) running search (which reads from the index) and
> posts (which writes to the index) transactions. After a while, we'd
> run out of file handles which froze Tomcat because it could no longer
> open sockets to accept incoming connections. Our sysadmin told me
> there were a bunch of orphaned file descriptors on files in /roller-
> index directory. I am not sure if that's the reason for Tomcat
> process running out of files but it seems likely. Has anybody ever
> experienced this problem with Roller search index?
>
> Max
>