You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2010/01/11 21:20:17 UTC
Compound File Default
Should we really still be defaulting to true for setUseCompoundFile? Do people still run out of file handles? If so, why not have them turn it on, instead of everyone else having to turn it off.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Michael McCandless <lu...@mikemccandless.com>.
+1
I think we should make it Version dependent...
Mike
On Mon, Jan 11, 2010 at 3:20 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Should we really still be defaulting to true for setUseCompoundFile? Do people still run out of file handles? If so, why not have them turn it on, instead of everyone else having to turn it off.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Otis Gospodnetic <ot...@yahoo.com>.
+1. I never liked having the compound format be the default, since increasing the max # of open file handles is a well documented thing, at least in the UNIX world.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Grant Ingersoll <gs...@apache.org>
> To: java-dev@lucene.apache.org
> Sent: Mon, January 11, 2010 3:20:17 PM
> Subject: Compound File Default
>
> Should we really still be defaulting to true for setUseCompoundFile? Do people
> still run out of file handles? If so, why not have them turn it on, instead of
> everyone else having to turn it off.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Jason Rutherglen <ja...@gmail.com>.
Maybe the default can be conditional on the platform like NIOFSDirectory.
On Mon, Jan 11, 2010 at 1:25 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
> On Mon, Jan 11, 2010 at 03:20:17PM -0500, Grant Ingersoll wrote:
>> Should we really still be defaulting to true for setUseCompoundFile? Do
>> people still run out of file handles?
>
> Yep. You're going to smack up against that limit pretty quick on Mac OS X:
>
> marvin@smokey:~ $ ulimit -n
> 256
>
>> If so, why not have them turn it on, instead of everyone else having to turn
>> it off.
>
> Can you up the file descriptor limit from within a running JVM?
>
> If not, you're setting yourself up with a non-portable default.
>
> Marvin Humphrey
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Earwin Burrfoot <ea...@gmail.com>.
256 here (MBP)
On Tue, Jan 12, 2010 at 17:49, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Jan 11, 2010, at 4:25 PM, Marvin Humphrey wrote:
>
>> On Mon, Jan 11, 2010 at 03:20:17PM -0500, Grant Ingersoll wrote:
>>> Should we really still be defaulting to true for setUseCompoundFile? Do
>>> people still run out of file handles?
>>
>> Yep. You're going to smack up against that limit pretty quick on Mac OS X:
>>
>> marvin@smokey:~ $ ulimit -n
>> 256
>
> My Mac (non-laptop) reports:
> ulimit -n
> 2560
>
> And I know I didn't change it.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 12, 2010, at 1:32 PM, Marvin Humphrey wrote:
>
> But beyond that, Lucene adopted the compound file format default for a reason,
> right? What's changed about the environment that justifies overturning that
> decision?
The history, as I recall, is it used to be off in 1.x. Then, b/c some people (not a lot) were hitting it, we turned on CFS by default. The perf. penalty is pretty significant and I think it is far easier to tell someone how to up their file handle limit or turn it on, then it is to overcome the sense that Lucene is slower than it really is.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Jason Rutherglen <ja...@gmail.com>.
I always turn CFS off because it's extra work (no payoff), how's it
possible to run into an out of fd limit with a merge factor of 10?
On Tue, Jan 12, 2010 at 2:35 PM, Mark Miller <ma...@gmail.com> wrote:
> Otis Gospodnetic wrote:
>> At the same time, seeing how some people benchmark systems without tuning them and then publish their results, cfs may be safer.
>>
>>
> Though at the same time you get nailed with a 10-15% indexing speed hit.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by John Wang <jo...@gmail.com>.
+1.
-John
On Tue, Jan 12, 2010 at 8:16 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Heh, yeah, I forgot about that. Pick the lesser evil? I like speedier
> defaults.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>
>
>
> ----- Original Message ----
> > From: Mark Miller <ma...@gmail.com>
> > To: java-dev@lucene.apache.org
> > Sent: Tue, January 12, 2010 5:35:49 PM
> > Subject: Re: Compound File Default
> >
> > Otis Gospodnetic wrote:
> > > At the same time, seeing how some people benchmark systems without
> tuning them
> > and then publish their results, cfs may be safer.
> > >
> > >
> > Though at the same time you get nailed with a 10-15% indexing speed hit.
> >
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Compound File Default
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Heh, yeah, I forgot about that. Pick the lesser evil? I like speedier defaults.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Mark Miller <ma...@gmail.com>
> To: java-dev@lucene.apache.org
> Sent: Tue, January 12, 2010 5:35:49 PM
> Subject: Re: Compound File Default
>
> Otis Gospodnetic wrote:
> > At the same time, seeing how some people benchmark systems without tuning them
> and then publish their results, cfs may be safer.
> >
> >
> Though at the same time you get nailed with a 10-15% indexing speed hit.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Mark Miller <ma...@gmail.com>.
Otis Gospodnetic wrote:
> At the same time, seeing how some people benchmark systems without tuning them and then publish their results, cfs may be safer.
>
>
Though at the same time you get nailed with a 10-15% indexing speed hit.
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I think what has changed is that a lot more people hit this problem, and a number of people provided answers, so it's much easier now for a new person to learn what to do when this limit is hit.
At the same time, seeing how some people benchmark systems without tuning them and then publish their results, cfs may be safer.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Marvin Humphrey <ma...@rectangular.com>
> To: java-dev@lucene.apache.org
> Sent: Tue, January 12, 2010 1:32:51 PM
> Subject: Re: Compound File Default
>
> On Tue, Jan 12, 2010 at 11:05:13AM -0500, Grant Ingersoll wrote:
> > At any rate, I feel pretty safe assuming no one is running a production
> > system on a MBP...
>
> I don't really care whether Lucene defaults to the compound file format or not
> (KS does, Lucy will, and that's good enough for me), but it seems weird that
> you're assuming that only Mac Book Pros have that default. Just for giggles,
> I checked my old PowerPC Mac Mini, running 10.5 -- it's got a limit of 256.
>
> But beyond that, Lucene adopted the compound file format default for a reason,
> right? What's changed about the environment that justifies overturning that
> decision?
>
> When I checked a RedHat 9 box several years ago, it was at 1024; when I
> checked a CentOS 5.2 box today, it was at 1024. A FreeBSD 5.3 box several
> years ago was at 65536. File descriptor limits don't seem to advance like
> hardware stats.
>
> Go ahead and change the default, but I've got a feeling you're about to
> relearn old lessons.
>
> Marvin Humphrey
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Jan 12, 2010 at 11:05:13AM -0500, Grant Ingersoll wrote:
> At any rate, I feel pretty safe assuming no one is running a production
> system on a MBP...
I don't really care whether Lucene defaults to the compound file format or not
(KS does, Lucy will, and that's good enough for me), but it seems weird that
you're assuming that only Mac Book Pros have that default. Just for giggles,
I checked my old PowerPC Mac Mini, running 10.5 -- it's got a limit of 256.
But beyond that, Lucene adopted the compound file format default for a reason,
right? What's changed about the environment that justifies overturning that
decision?
When I checked a RedHat 9 box several years ago, it was at 1024; when I
checked a CentOS 5.2 box today, it was at 1024. A FreeBSD 5.3 box several
years ago was at 65536. File descriptor limits don't seem to advance like
hardware stats.
Go ahead and change the default, but I've got a feeling you're about to
relearn old lessons.
Marvin Humphrey
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by DM Smith <dm...@gmail.com>.
I'm not sure that it's safe to assume that production use of Lucene is
not on a laptop or that it is always on big iron.
It makes sense that Lucene is embedded in all sorts of desktop
applications that might run on small machines. That certainly describes
the application that I work on.
I'm a bit leery of what is looking to me like Version hell. For example,
I need one version to ensure backward compatibility wrt file handles,
another wrt analysis, another wrt ....
As long as there is a clear way to (un)set the feature and it is
documented well, I'd skip the Version mechanism on an upgrade.
-- DM
On 01/12/2010 11:05 AM, Grant Ingersoll wrote:
> My MBP has 7168.
>
> Maybe something like MySQL or other tools modify it, but I'm pretty positive I didn't.
>
> At any rate, I feel pretty safe assuming no one is running a production system on a MBP...
>
> I suppose if we wanted to get really fancy, we could, on *NIX systems, exec ulimit and parse the results and then choose the appropriate setting. I personally think, however, that isn't time well spent and would just as soon have it set to false and tied to the Version.
>
>
> On Jan 12, 2010, at 10:09 AM, Marvin Humphrey wrote:
>
>
>> On Tue, Jan 12, 2010 at 09:49:09AM -0500, Grant Ingersoll wrote:
>>
>>> My Mac (non-laptop) reports:
>>> ulimit -n
>>> 2560
>>>
>>> And I know I didn't change it.
>>>
>> Before I posted, I had a few officemates corroborate. 4 people had 256 --
>> three on 10.6 and me on 10.5. I think these were all Mac Book Pros. The
>> exception was our DBA, who had high numbers (thousands) on both his MBP and
>> his desktop.
>>
>> Marvin Humphrey
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Grant Ingersoll <gr...@gmail.com>.
My MBP has 7168.
Maybe something like MySQL or other tools modify it, but I'm pretty positive I didn't.
At any rate, I feel pretty safe assuming no one is running a production system on a MBP...
I suppose if we wanted to get really fancy, we could, on *NIX systems, exec ulimit and parse the results and then choose the appropriate setting. I personally think, however, that isn't time well spent and would just as soon have it set to false and tied to the Version.
On Jan 12, 2010, at 10:09 AM, Marvin Humphrey wrote:
> On Tue, Jan 12, 2010 at 09:49:09AM -0500, Grant Ingersoll wrote:
>> My Mac (non-laptop) reports:
>> ulimit -n
>> 2560
>>
>> And I know I didn't change it.
>
> Before I posted, I had a few officemates corroborate. 4 people had 256 --
> three on 10.6 and me on 10.5. I think these were all Mac Book Pros. The
> exception was our DBA, who had high numbers (thousands) on both his MBP and
> his desktop.
>
> Marvin Humphrey
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Jan 12, 2010 at 09:49:09AM -0500, Grant Ingersoll wrote:
> My Mac (non-laptop) reports:
> ulimit -n
> 2560
>
> And I know I didn't change it.
Before I posted, I had a few officemates corroborate. 4 people had 256 --
three on 10.6 and me on 10.5. I think these were all Mac Book Pros. The
exception was our DBA, who had high numbers (thousands) on both his MBP and
his desktop.
Marvin Humphrey
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 11, 2010, at 4:25 PM, Marvin Humphrey wrote:
> On Mon, Jan 11, 2010 at 03:20:17PM -0500, Grant Ingersoll wrote:
>> Should we really still be defaulting to true for setUseCompoundFile? Do
>> people still run out of file handles?
>
> Yep. You're going to smack up against that limit pretty quick on Mac OS X:
>
> marvin@smokey:~ $ ulimit -n
> 256
My Mac (non-laptop) reports:
ulimit -n
2560
And I know I didn't change it.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Compound File Default
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Mon, Jan 11, 2010 at 03:20:17PM -0500, Grant Ingersoll wrote:
> Should we really still be defaulting to true for setUseCompoundFile? Do
> people still run out of file handles?
Yep. You're going to smack up against that limit pretty quick on Mac OS X:
marvin@smokey:~ $ ulimit -n
256
> If so, why not have them turn it on, instead of everyone else having to turn
> it off.
Can you up the file descriptor limit from within a running JVM?
If not, you're setting yourself up with a non-portable default.
Marvin Humphrey
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org