Forked files? was: RE: Too many open files?

[Scott Ganyo <sc...@eTapestry.com>]

Another idea to address this (quite common) problem:

Does anyone know if there are any Java file implementations that support a
"forked" file or a file with multiple streams?  Or, if not, do you know of
any design patterns or documents explaining the theory and design in this
kind of thing?  It would seem that if there was an efficient implementation
of a forked file, perhaps that could be used instead of the set of files
that Lucene currently uses to represent a segment.

Scott

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Tuesday, July 23, 2002 10:13 AM
> To: 'Lucene Users List'
> Subject: RE: Too many open files?
> 
> 
> Are you closing the searcher after each when done?
> 
> No: Waiting for the garbage collector is not a good idea.
> 
> Yes: It could be a timeout on the OS holding the files handles.
> 
> Either way, the only real option is to avoid thrashing the 
> searchers...
> 
> Scott
> 
> > -----Original Message-----
> > From: Hang Li [mailto:hxl@careersite.com]
> > Sent: Tuesday, July 23, 2002 10:10 AM
> > To: Lucene Users List
> > Subject: Re: Too many open files?
> > 
> > 
> > Thanks for your quick reponse, I still want to know why we 
> ran out of
> > file descriptors.
> > 
> > --Yup.  Cache and reuse your Searcher as much as possible.
> > 
> > --Scott
> > 
> > > -----Original Message-----
> > > From: Hang Li [mailto:hxl@careersite.com]
> > > Sent: Tuesday, July 23, 2002 9:59 AM
> > > To: Lucene Users List
> > > Subject: Too many open files?
> > >
> > >
> > > >
> > >
> > > I have seen a lot postings about this topic. Any final thoughts?
> > >
> > > We did a simple stress test, Lucene would produce this error
> > > between 30 - 80
> > > concurren searches.  The index directory has 24 files (15 
> > fields), and
> > 
> > > "
> > > ulimit -n
> > > 32768
> > > ",
> > >
> > > there should be more than enough FDs.  Note, we did not do
> > > any writings to index
> > > while we were searching.  Any ideas? Thx.
> > >
> > 
> > 
> > 
> > --
> > To unsubscribe, e-mail:   
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>