You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by root <am...@dkf.de> on 2002/04/09 14:02:29 UTC
too many open files in system
Hi List!
Doesn't Lucene releases the filehandles??
because I get "too many open files in system" after running lucene a while!
I use the 1.2 rc 4 version!
regards
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by petite_abeille <pe...@mac.com>.
> how many open files you think can be used at your process??
Not sure. It varies with usage pattern. I will check it out in any case.
> cat /proc/sys/fs/file-max
cat: /proc/sys/fs/file-max: No such file or directory
> echo 50000 > /proc/sys/fs/file-max
Unfortunately, I cannot use this kind of "quick fix" as my app is a
desktop app and can access the user account only.
PA.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by mario martinez gomez <ma...@newknow.com>.
petite_abeille wrote:
>> On Tuesday, 9. April 2002 14:08, you wrote:
>>
>>> root wrote:
>>>
>>>> Doesn't Lucene releases the filehandles??
>>>>
>>>> because I get "too many open files in system" after running lucene a
>>>> while!
>>>
>>>
>>> Are you closing the readers and writers after you've finished using
>>> them?
>>>
>>> cheers,
>>>
>>> Chris
>>
>>
>>
>> Yes I close the readers and writers!
>
>
>
> By the way, did you ever solved this problem? I want through that
> thread and everybody seem to be passing the buck to somebody else... :-(
>
> PA.
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
how many open files you think can be used at your process??
look at the max-file field at proc to be sure its enough
maybe this could help ....... or maybe not
cat /proc/sys/fs/file-max
if you need more:
echo 50000 > /proc/sys/fs/file-max
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by petite_abeille <pe...@mac.com>.
> On Tuesday, 9. April 2002 14:08, you wrote:
>> root wrote:
>>> Doesn't Lucene releases the filehandles??
>>>
>>> because I get "too many open files in system" after running lucene a
>>> while!
>>
>> Are you closing the readers and writers after you've finished using
>> them?
>>
>> cheers,
>>
>> Chris
>
>
> Yes I close the readers and writers!
By the way, did you ever solved this problem? I want through that thread
and everybody seem to be passing the buck to somebody else... :-(
PA.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by Karl Øie <ka...@gan.no>.
I have worked a little with the cocoon indexer and it indexes each
xml-attribute in a Field. I have done some indexing on both plaintext and xml
sources and i think the "Too many open files" problem is directly related to
number of fields stored in a document in a index.
the reason for this is that i have never encountered "Too many open files"
when indexing clean text into one large field, but when creating many-many
fields as required by indexing xml i got a "Too many open files" until i had
to use a ram-dir to index document batches into..
mvh karl øie
On Tuesday 09 April 2002 16:42, you wrote:
> This sounds like a question for Cocoon people, as what you are asking
> about seems to be related to Cocoon's usage of Lucene, not the core
> Lucene API.
>
> Otis
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I've indexed 250,000 items (database rows, not files) with Lucene on a
system like this:
[otis@kyle blink]$ ulimit
unlimited
[otis@kyle blink]$ tcsh
> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 1000000 kbytes
memoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 8192
openfiles 1024
I didn't encounter the 'too many open files' error.
You could change your code to optimize index after adding every X
documents to the index.
If you add Documents to a RAMDirectory-based IndexWriter, and them use
IndexWriter's addIndexes method, you will have an optimized index after
calling addIndexes method, plus you'll speed up indexing.
Otis
--- "Nader S. Henein" <ns...@bayt.net> wrote:
> The issue is the same with Lucene when you index, if you're
> indexing 200 000 files the amount of files created by the index cause
> the
> index system to run out of file handles, is there an equation to find
> out
> how many files will be created by the indexer based on the number of
> files
> we want indexed ? , becasue that would give me an idea of how many
> file
> handles
> I would need
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, April 09, 2002 6:42 PM
> To: Lucene Users List; ian@plusfour.org
> Subject: RE: too many open files in system
>
>
> This sounds like a question for Cocoon people, as what you are asking
> about seems to be related to Cocoon's usage of Lucene, not the core
> Lucene API.
>
> Otis
>
> --- Ian Forsyth <ia...@plusfour.org> wrote:
> > I'm calling in response to the LuceneCocconIndexer, is this class
> an
> > XML
> > file indexer? (excuse my ignorance i am just stepping into this
> whole
> > thing..)
> >
> > I do a lot of development with PHP, on different platforms
> > (WIN,*NIXES) and
> > I want to get into indexing data...
> >
> > I am wondering if there are classes existing that index XML files,
> > and for
> > search results, spit out an xml stream..
> >
> > I am figuring I would send results to lucene via the cmd line, but
> i
> > am not
> > sure how i would capture the xml results.. Do I have to have Lucene
> > make a
> > physical XML file called for instance searchresults.xml then i set
> my
> > XSLT
> > parser to that file.. ?
> >
> > I am very curious about this whole thing and any help would be a
> > super help.
> >
> > Regards,
> > Ian
> >
> > -----Original Message-----
> > From: Nader S. Henein [mailto:nsh@bayt.net]
> > Sent: Tuesday, April 09, 2002 8:56 AM
> > To: Lucene Users List; amayen@dkf.de
> > Subject: RE: too many open files in system
> >
> >
> > that depends on how many files you're indexing .. I still have to
> > figure out
> > too what logic does the LuceneCocoonIndexer adhere when it is
> > creating the
> > index files
> >
> >
> > -----Original Message-----
> > From: root [mailto:amayen@dkf.de]
> > Sent: Tuesday, April 09, 2002 4:50 PM
> > To: Lucene Users List
> > Subject: Re: too many open files in system
> >
> >
> > On Tuesday, 9. April 2002 14:08, you wrote:
> > > root wrote:
> > > > Doesn't Lucene releases the filehandles??
> > > >
> > > > because I get "too many open files in system" after running
> > lucene a
> > > > while!
> > >
> > > Are you closing the readers and writers after you've finished
> using
> > them?
> > >
> > > cheers,
> > >
> > > Chris
> >
> >
> > Yes I close the readers and writers!
> >
> >
> > @Nader S. Henein
> >
> > If I increase the filehandles, to what count should I increase
> them?
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Tax Center - online filing with TurboTax
> http://taxes.yahoo.com/
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by "Nader S. Henein" <ns...@bayt.net>.
The issue is the same with Lucene when you index, if you're
indexing 200 000 files the amount of files created by the index cause the
index system to run out of file handles, is there an equation to find out
how many files will be created by the indexer based on the number of files
we want indexed ? , becasue that would give me an idea of how many file
handles
I would need
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Tuesday, April 09, 2002 6:42 PM
To: Lucene Users List; ian@plusfour.org
Subject: RE: too many open files in system
This sounds like a question for Cocoon people, as what you are asking
about seems to be related to Cocoon's usage of Lucene, not the core
Lucene API.
Otis
--- Ian Forsyth <ia...@plusfour.org> wrote:
> I'm calling in response to the LuceneCocconIndexer, is this class an
> XML
> file indexer? (excuse my ignorance i am just stepping into this whole
> thing..)
>
> I do a lot of development with PHP, on different platforms
> (WIN,*NIXES) and
> I want to get into indexing data...
>
> I am wondering if there are classes existing that index XML files,
> and for
> search results, spit out an xml stream..
>
> I am figuring I would send results to lucene via the cmd line, but i
> am not
> sure how i would capture the xml results.. Do I have to have Lucene
> make a
> physical XML file called for instance searchresults.xml then i set my
> XSLT
> parser to that file.. ?
>
> I am very curious about this whole thing and any help would be a
> super help.
>
> Regards,
> Ian
>
> -----Original Message-----
> From: Nader S. Henein [mailto:nsh@bayt.net]
> Sent: Tuesday, April 09, 2002 8:56 AM
> To: Lucene Users List; amayen@dkf.de
> Subject: RE: too many open files in system
>
>
> that depends on how many files you're indexing .. I still have to
> figure out
> too what logic does the LuceneCocoonIndexer adhere when it is
> creating the
> index files
>
>
> -----Original Message-----
> From: root [mailto:amayen@dkf.de]
> Sent: Tuesday, April 09, 2002 4:50 PM
> To: Lucene Users List
> Subject: Re: too many open files in system
>
>
> On Tuesday, 9. April 2002 14:08, you wrote:
> > root wrote:
> > > Doesn't Lucene releases the filehandles??
> > >
> > > because I get "too many open files in system" after running
> lucene a
> > > while!
> >
> > Are you closing the readers and writers after you've finished using
> them?
> >
> > cheers,
> >
> > Chris
>
>
> Yes I close the readers and writers!
>
>
> @Nader S. Henein
>
> If I increase the filehandles, to what count should I increase them?
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This sounds like a question for Cocoon people, as what you are asking
about seems to be related to Cocoon's usage of Lucene, not the core
Lucene API.
Otis
--- Ian Forsyth <ia...@plusfour.org> wrote:
> I'm calling in response to the LuceneCocconIndexer, is this class an
> XML
> file indexer? (excuse my ignorance i am just stepping into this whole
> thing..)
>
> I do a lot of development with PHP, on different platforms
> (WIN,*NIXES) and
> I want to get into indexing data...
>
> I am wondering if there are classes existing that index XML files,
> and for
> search results, spit out an xml stream..
>
> I am figuring I would send results to lucene via the cmd line, but i
> am not
> sure how i would capture the xml results.. Do I have to have Lucene
> make a
> physical XML file called for instance searchresults.xml then i set my
> XSLT
> parser to that file.. ?
>
> I am very curious about this whole thing and any help would be a
> super help.
>
> Regards,
> Ian
>
> -----Original Message-----
> From: Nader S. Henein [mailto:nsh@bayt.net]
> Sent: Tuesday, April 09, 2002 8:56 AM
> To: Lucene Users List; amayen@dkf.de
> Subject: RE: too many open files in system
>
>
> that depends on how many files you're indexing .. I still have to
> figure out
> too what logic does the LuceneCocoonIndexer adhere when it is
> creating the
> index files
>
>
> -----Original Message-----
> From: root [mailto:amayen@dkf.de]
> Sent: Tuesday, April 09, 2002 4:50 PM
> To: Lucene Users List
> Subject: Re: too many open files in system
>
>
> On Tuesday, 9. April 2002 14:08, you wrote:
> > root wrote:
> > > Doesn't Lucene releases the filehandles??
> > >
> > > because I get "too many open files in system" after running
> lucene a
> > > while!
> >
> > Are you closing the readers and writers after you've finished using
> them?
> >
> > cheers,
> >
> > Chris
>
>
> Yes I close the readers and writers!
>
>
> @Nader S. Henein
>
> If I increase the filehandles, to what count should I increase them?
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by Ian Forsyth <ia...@plusfour.org>.
I'm calling in response to the LuceneCocconIndexer, is this class an XML
file indexer? (excuse my ignorance i am just stepping into this whole
thing..)
I do a lot of development with PHP, on different platforms (WIN,*NIXES) and
I want to get into indexing data...
I am wondering if there are classes existing that index XML files, and for
search results, spit out an xml stream..
I am figuring I would send results to lucene via the cmd line, but i am not
sure how i would capture the xml results.. Do I have to have Lucene make a
physical XML file called for instance searchresults.xml then i set my XSLT
parser to that file.. ?
I am very curious about this whole thing and any help would be a super help.
Regards,
Ian
-----Original Message-----
From: Nader S. Henein [mailto:nsh@bayt.net]
Sent: Tuesday, April 09, 2002 8:56 AM
To: Lucene Users List; amayen@dkf.de
Subject: RE: too many open files in system
that depends on how many files you're indexing .. I still have to figure out
too what logic does the LuceneCocoonIndexer adhere when it is creating the
index files
-----Original Message-----
From: root [mailto:amayen@dkf.de]
Sent: Tuesday, April 09, 2002 4:50 PM
To: Lucene Users List
Subject: Re: too many open files in system
On Tuesday, 9. April 2002 14:08, you wrote:
> root wrote:
> > Doesn't Lucene releases the filehandles??
> >
> > because I get "too many open files in system" after running lucene a
> > while!
>
> Are you closing the readers and writers after you've finished using them?
>
> cheers,
>
> Chris
Yes I close the readers and writers!
@Nader S. Henein
If I increase the filehandles, to what count should I increase them?
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by "Nader S. Henein" <ns...@bayt.net>.
that depends on how many files you're indexing .. I still have to figure out
too what logic does the LuceneCocoonIndexer adhere when it is creating the
index files
-----Original Message-----
From: root [mailto:amayen@dkf.de]
Sent: Tuesday, April 09, 2002 4:50 PM
To: Lucene Users List
Subject: Re: too many open files in system
On Tuesday, 9. April 2002 14:08, you wrote:
> root wrote:
> > Doesn't Lucene releases the filehandles??
> >
> > because I get "too many open files in system" after running lucene a
> > while!
>
> Are you closing the readers and writers after you've finished using them?
>
> cheers,
>
> Chris
Yes I close the readers and writers!
@Nader S. Henein
If I increase the filehandles, to what count should I increase them?
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by root <am...@dkf.de>.
On Tuesday, 9. April 2002 14:08, you wrote:
> root wrote:
> > Doesn't Lucene releases the filehandles??
> >
> > because I get "too many open files in system" after running lucene a
> > while!
>
> Are you closing the readers and writers after you've finished using them?
>
> cheers,
>
> Chris
Yes I close the readers and writers!
@Nader S. Henein
If I increase the filehandles, to what count should I increase them?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by Chris Withers <ch...@nipltd.com>.
root wrote:
>
> Doesn't Lucene releases the filehandles??
>
> because I get "too many open files in system" after running lucene a while!
Are you closing the readers and writers after you've finished using them?
cheers,
Chris
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: too many open files in system
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Judging from other messages in this thread it seems that the cause of
your problem could be an unoptimized index (somebody said that lots of
files need to be opened for searches).
Try optimizing your index. Optimizing an index will reduce the number
of files comprising your index.
Otis
--- root <am...@dkf.de> wrote:
> Hi List!
>
> Doesn't Lucene releases the filehandles??
>
> because I get "too many open files in system" after running lucene a
> while!
>
> I use the 1.2 rc 4 version!
>
>
> regards
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: too many open files in system
Posted by "Nader S. Henein" <ns...@bayt.net>.
it's not a matter of releasing the handles, it needs to keep them open,
this tricked me as well I thought it kept the file handles of the
source XML files open, but if you look at the code it actually reads the
contents of
the files from an HTTP request, the file handles are consumed by the files
that lucene creates
to store the index results, that's why you get the same error when you try
to search as well
it tries to open all the files but runs out of handles in the process, you
have to increase your
unix file handles and reboot the system (how to depends on your OS), this
solves one problem.
I just hit another one, but I'm convinced it's worth it, I've gotten
excellent results after indexing
20 000 files, very fast and very responsive and if it's going to take some
tweaking to get it over this
problem so be it, that's the joy of open source
cheers .. I hope that was useful
-----Original Message-----
From: root [mailto:amayen@dkf.de]
Sent: Tuesday, April 09, 2002 4:02 PM
To: lucene-user@jakarta.apache.org
Subject: too many open files in system
Hi List!
Doesn't Lucene releases the filehandles??
because I get "too many open files in system" after running lucene a while!
I use the 1.2 rc 4 version!
regards
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>