You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Dawid Weiss <da...@gmail.com> on 2012/04/16 13:46:57 UTC
Open files exceeded, freebsd.
I think this may be an overall limit of open files on the machine -- look:
[root@lucene /usr/home/dweiss]# sysctl kern.maxfiles
kern.maxfiles: 12328
[root@lucene /usr/home/dweiss]# sysctl kern.maxfilesperproc
kern.maxfilesperproc: 11095
[root@lucene /usr/home/dweiss]# sysctl kern.openfiles
kern.openfiles: 10507
So it's close to the limit of maxfiles. Also, other processes fail
with file handles as well (grep "Too many" var/log/messages).
I have zero knowledge of freebsd administration. Can this be topped a
bit? This doesn't seem to do the trick:
[root@lucene /usr/home/dweiss]# sysctl kern.maxfiles=40000
kern.maxfiles: 12328
sysctl: kern.maxfiles: Operation not permitted
This:
http://forums.freebsd.org/showthread.php?t=13632
mentions loader.conf but I don't want to fiddle with something that
can bring down the machine :)
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Open files exceeded, freebsd.
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Apr 16, 2012 at 4:25 PM, Dawid Weiss
<da...@cs.put.poznan.pl> wrote:
> We've established with @infra that this is caused by another jail
> eating up 8k descriptors. They're told to limit resource usage and
> after that jenkins should go back to normal.
>
Thanks for the help Dawid... sure was bad luck to happen just as you
committed the new test stuff.
I spent some time (after fixing a few problems last night) debugging
maximum file usage while running tests.
In general we have really no serious leaks, there are some
onesy-twoesy that we already know about,
like not being able to close the SpellChecker searcher in the solr
IndexBasedSpellChecker test, but leaks are really no issue.
Still, lucene-core tests on average (with 4 jvms) use about 300-400
open files at any point, and a few tests bring this up temporarily
to even close to 1000. So its still good to keep an eye on, I think
these indexing tests in lucene are really the only ones that might do
this,
especially if they get crazy configurations.
So long term i still think we should implement something along the
lines of https://issues.apache.org/jira/browse/LUCENE-3195, at least
so if a test does something truly absurd here as far as files we know
about it and can adjust the parameters appropriately: and can fix the
worst tests so they choose parameters that avoid this
[_TestUtil.reduceOpenFiles(IndexWriter)]. Otherwise, infra will be
banging on our door next time :)
--
lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Open files exceeded, freebsd.
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
We've established with @infra that this is caused by another jail
eating up 8k descriptors. They're told to limit resource usage and
after that jenkins should go back to normal.
Dawid
On Mon, Apr 16, 2012 at 1:56 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> The problem is that this is a Jail, we only have "fake" root access.
>
> Uwe
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
>> Sent: Monday, April 16, 2012 1:47 PM
>> To: dev@lucene.apache.org
>> Subject: Open files exceeded, freebsd.
>>
>> I think this may be an overall limit of open files on the machine -- look:
>>
>> [root@lucene /usr/home/dweiss]# sysctl kern.maxfiles
>> kern.maxfiles: 12328
>> [root@lucene /usr/home/dweiss]# sysctl kern.maxfilesperproc
>> kern.maxfilesperproc: 11095
>> [root@lucene /usr/home/dweiss]# sysctl kern.openfiles
>> kern.openfiles: 10507
>>
>> So it's close to the limit of maxfiles. Also, other processes fail with file handles
>> as well (grep "Too many" var/log/messages).
>>
>> I have zero knowledge of freebsd administration. Can this be topped a bit? This
>> doesn't seem to do the trick:
>>
>> [root@lucene /usr/home/dweiss]# sysctl kern.maxfiles=40000
>> kern.maxfiles: 12328
>> sysctl: kern.maxfiles: Operation not permitted
>>
>> This:
>> http://forums.freebsd.org/showthread.php?t=13632
>>
>> mentions loader.conf but I don't want to fiddle with something that can bring
>> down the machine :)
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Open files exceeded, freebsd.
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
The problem is that this is a Jail, we only have "fake" root access.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Sent: Monday, April 16, 2012 1:47 PM
> To: dev@lucene.apache.org
> Subject: Open files exceeded, freebsd.
>
> I think this may be an overall limit of open files on the machine -- look:
>
> [root@lucene /usr/home/dweiss]# sysctl kern.maxfiles
> kern.maxfiles: 12328
> [root@lucene /usr/home/dweiss]# sysctl kern.maxfilesperproc
> kern.maxfilesperproc: 11095
> [root@lucene /usr/home/dweiss]# sysctl kern.openfiles
> kern.openfiles: 10507
>
> So it's close to the limit of maxfiles. Also, other processes fail with file handles
> as well (grep "Too many" var/log/messages).
>
> I have zero knowledge of freebsd administration. Can this be topped a bit? This
> doesn't seem to do the trick:
>
> [root@lucene /usr/home/dweiss]# sysctl kern.maxfiles=40000
> kern.maxfiles: 12328
> sysctl: kern.maxfiles: Operation not permitted
>
> This:
> http://forums.freebsd.org/showthread.php?t=13632
>
> mentions loader.conf but I don't want to fiddle with something that can bring
> down the machine :)
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org