You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Dvora <ba...@gmail.com> on 2010/01/07 09:32:30 UTC
Is there a way to limit the size of an index?
Hi all,
According to the FAQ, "An even more complex and optimal solution:
Write a version of FSDirectory that, when a file exceeds 2GB, creates a
subdirectory and represents the file as a series of files."
Is that solution implemented already? If not, can you guide me please how
can I achieve that? Is this solution requires changes in the Reader as well?
Thanks.
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27056573.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Jan 8, 2010 at 1:22 AM, Babak Farhang <fa...@gmail.com> wrote:
>>> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
>
> How about using the *able* moniker to make it clear we're referring to
> the size of the to-be-merged segment, not the resultant merged
> segment? I.e. naming it something like "maxMergeableSegSizeMB" ..
I like this the best so far! It's clear that it applies to the
segments to-be-merged, not the final merged segments.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Babak Farhang <fa...@gmail.com>.
>> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
How about using the *able* moniker to make it clear we're referring to
the size of the to-be-merged segment, not the resultant merged
segment? I.e. naming it something like "maxMergeableSegSizeMB" ..
On Thu, Jan 7, 2010 at 3:21 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> The naming is unclear, when I looked at this I had to thumb
> through the code a fair bit before discerning if it was the
> input segments or the output segment of a merge (it's the
> former). Though I find the current functionality somewhat odd
> because it will inherently exceed the given size with a merge,
> which is uncontrollable, making the exact setting somewhat
> arbitrary... What was the original use case?
>
> On Thu, Jan 7, 2010 at 11:23 AM, Otis Gospodnetic
> <ot...@yahoo.com> wrote:
>>> Merge factor controls how many segments are merged at once. The default is 10.
>>>
>>> The maxMergeMB setting sets the max size for a given segment to be
>>> included in a merge.
>>
>> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>>
>>> Roughly, the upper bound on merged segments is the sum of their sizes.
>>>
>>> So the rough upper bound on any segment's size is mergeFactor * maxMergeMB.
>>>
>>> Mike
>>>
>>> On Thu, Jan 7, 2010 at 11:04 AM, Dvora wrote:
>>> >
>>> > Can you explain how the combination of merge factor and max merge size
>>> > control the size of files?
>>> >
>>> > For example, if one would like to limit the files size to 3,4 or 7MB - how
>>> > these parameters values can be predicted?
>>> >
>>> >
>>> >
>>> > Michael McCandless-2 wrote:
>>> >>
>>> >>
>>> >> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
>>> >> in size. With a default merge factor of 10, this should "generally"
>>> >> mean you don't get a segment over 10MB, though it may not be a hard
>>> >> guarantee (you can lower the 1.0 if you still see a segment over 10
>>> >> MB).
>>> >>
>>> >>
>>> >
>>> > --
>>> > View this message in context:
>>> http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
>>> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Jason Rutherglen <ja...@gmail.com>.
The naming is unclear, when I looked at this I had to thumb
through the code a fair bit before discerning if it was the
input segments or the output segment of a merge (it's the
former). Though I find the current functionality somewhat odd
because it will inherently exceed the given size with a merge,
which is uncontrollable, making the exact setting somewhat
arbitrary... What was the original use case?
On Thu, Jan 7, 2010 at 11:23 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>> Merge factor controls how many segments are merged at once. The default is 10.
>>
>> The maxMergeMB setting sets the max size for a given segment to be
>> included in a merge.
>
> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>
>> Roughly, the upper bound on merged segments is the sum of their sizes.
>>
>> So the rough upper bound on any segment's size is mergeFactor * maxMergeMB.
>>
>> Mike
>>
>> On Thu, Jan 7, 2010 at 11:04 AM, Dvora wrote:
>> >
>> > Can you explain how the combination of merge factor and max merge size
>> > control the size of files?
>> >
>> > For example, if one would like to limit the files size to 3,4 or 7MB - how
>> > these parameters values can be predicted?
>> >
>> >
>> >
>> > Michael McCandless-2 wrote:
>> >>
>> >>
>> >> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
>> >> in size. With a default merge factor of 10, this should "generally"
>> >> mean you don't get a segment over 10MB, though it may not be a hard
>> >> guarantee (you can lower the 1.0 if you still see a segment over 10
>> >> MB).
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
>> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Kay Kay <ka...@gmail.com>.
On 1/7/10 11:23 AM, Otis Gospodnetic wrote:
>> Merge factor controls how many segments are merged at once. The default is 10.
>>
>> The maxMergeMB setting sets the max size for a given segment to be
>> included in a merge.
>>
> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
>
Personally, +1 on the new naming , as it seems self explanatory than
maxMergeMB .
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>
>
>> Roughly, the upper bound on merged segments is the sum of their sizes.
>>
>> So the rough upper bound on any segment's size is mergeFactor * maxMergeMB.
>>
>> Mike
>>
>> On Thu, Jan 7, 2010 at 11:04 AM, Dvora wrote:
>>
>>> Can you explain how the combination of merge factor and max merge size
>>> control the size of files?
>>>
>>> For example, if one would like to limit the files size to 3,4 or 7MB - how
>>> these parameters values can be predicted?
>>>
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>
>>>>
>>>> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
>>>> in size. With a default merge factor of 10, this should "generally"
>>>> mean you don't get a segment over 10MB, though it may not be a hard
>>>> guarantee (you can lower the 1.0 if you still see a segment over 10
>>>> MB).
>>>>
>>>>
>>>>
>>> --
>>> View this message in context:
>>>
>> http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
>>
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Sure, sounds good, maybe even drop "ing".
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: Michael McCandless <lu...@mikemccandless.com>
> To: java-user@lucene.apache.org
> Sent: Thu, January 7, 2010 2:28:15 PM
> Subject: Re: Is there a way to limit the size of an index?
>
> On Thu, Jan 7, 2010 at 2:23 PM, Otis Gospodnetic
> wrote:
> >> Merge factor controls how many segments are merged at once. The default is
> 10.
> >>
> >> The maxMergeMB setting sets the max size for a given segment to be
> >> included in a merge.
> >
> > I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what
> this does?
>
> Well... that setting is already in LogByteSizeMergePolicy (not
> IndexWriter), so I think in that context it's pretty clear?
>
> Though I'd love to find a better name that conveys that the size
> limitation applies to the segments *being* merged, not to the
> resulting merged segment. maxStartingSegSizeMB?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Jan 7, 2010 at 2:23 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>> Merge factor controls how many segments are merged at once. The default is 10.
>>
>> The maxMergeMB setting sets the max size for a given segment to be
>> included in a merge.
>
> I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
Well... that setting is already in LogByteSizeMergePolicy (not
IndexWriter), so I think in that context it's pretty clear?
Though I'd love to find a better name that conveys that the size
limitation applies to the segments *being* merged, not to the
resulting merged segment. maxStartingSegSizeMB?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
> Merge factor controls how many segments are merged at once. The default is 10.
>
> The maxMergeMB setting sets the max size for a given segment to be
> included in a merge.
I wonder if renaming that to maxSegSizeMergeMB would make it more obvious what this does?
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
> Roughly, the upper bound on merged segments is the sum of their sizes.
>
> So the rough upper bound on any segment's size is mergeFactor * maxMergeMB.
>
> Mike
>
> On Thu, Jan 7, 2010 at 11:04 AM, Dvora wrote:
> >
> > Can you explain how the combination of merge factor and max merge size
> > control the size of files?
> >
> > For example, if one would like to limit the files size to 3,4 or 7MB - how
> > these parameters values can be predicted?
> >
> >
> >
> > Michael McCandless-2 wrote:
> >>
> >>
> >> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
> >> in size. With a default merge factor of 10, this should "generally"
> >> mean you don't get a segment over 10MB, though it may not be a hard
> >> guarantee (you can lower the 1.0 if you still see a segment over 10
> >> MB).
> >>
> >>
> >
> > --
> > View this message in context:
> http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Sun, Jan 10, 2010 at 7:33 AM, Dvora <ba...@gmail.com> wrote:
>
> I'm storing and reading the documents using Compass, not Lucene directly. I
> didn't touch those parameters, so I guess the default values are being used
> (I do see cfs files in the index).
OK. If your index directory has *.cfs files, then you are using the
default compound file format (I'm not sure whether Compass changes
that default either).
> How the ramBufferSizeMB parameter affect the files size? What value should I
> use in order to have 6MB files?
ramBufferSizeMB controls how big the initially created segments are.
Just how big a segment you get for a given ramBufferMB is very app
dependent, because the RAM efficiency of IndexWriter depends on things
like whether you have many unique terms (= worse RAM efficiency).
Generally larger RAM buffers have better efficiency.
Merging can only create bigger segments from those flushed segments.
So, you have to ensure ramBufferSizeMB is set such that in your use
case it never up and flushes a segment bigger than your 10 MB size
limit.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
I'm storing and reading the documents using Compass, not Lucene directly. I
didn't touch those parameters, so I guess the default values are being used
(I do see cfs files in the index).
How the ramBufferSizeMB parameter affect the files size? What value should I
use in order to have 6MB files?
Michael McCandless-2 wrote:
>
> What did you set your IndexWriter ramBufferSizeMB to? That controls
> how large the initial segments are.
>
> Are you using compound file format? (that's the default).
>
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27097887.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
What did you set your IndexWriter ramBufferSizeMB to? That controls
how large the initial segments are.
Are you using compound file format? (that's the default).
Mike
On Sun, Jan 10, 2010 at 7:03 AM, Dvora <ba...@gmail.com> wrote:
>
> Oh, as an excercise I tried to create 6MB files. Using the rule mentioned
> before, I set the maxMergeMB to 0.6 (and then 0.62, 06.64... 1.8) and used
> the default mergeFactor - I thought that should do for 6MB files...
>
>
> Michael McCandless-2 wrote:
>>
>> What settings did you use (mergeFactor, maxMergeMB)?
>>
>> "nothing in the middle" is normal. Segments sizes are often rather
>> quantized, for a newly created index. Ie a bunch of segs will be 1
>> MB, a bunch ~8, a bunch ~64 (assuming the IW flushes at ~ 1MB and
>> mergeFactor is 8).
>>
>> Mike
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27097693.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
Oh, as an excercise I tried to create 6MB files. Using the rule mentioned
before, I set the maxMergeMB to 0.6 (and then 0.62, 06.64... 1.8) and used
the default mergeFactor - I thought that should do for 6MB files...
Michael McCandless-2 wrote:
>
> What settings did you use (mergeFactor, maxMergeMB)?
>
> "nothing in the middle" is normal. Segments sizes are often rather
> quantized, for a newly created index. Ie a bunch of segs will be 1
> MB, a bunch ~8, a bunch ~64 (assuming the IW flushes at ~ 1MB and
> mergeFactor is 8).
>
> Mike
>
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27097693.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
What settings did you use (mergeFactor, maxMergeMB)?
"nothing in the middle" is normal. Segments sizes are often rather
quantized, for a newly created index. Ie a bunch of segs will be 1
MB, a bunch ~8, a bunch ~64 (assuming the IW flushes at ~ 1MB and
mergeFactor is 8).
Mike
On Sat, Jan 9, 2010 at 3:12 PM, Dvora <ba...@gmail.com> wrote:
>
> Well, this rule seems not working...
>
> I tried to create an index of 90k documents, with different merge factors.
> Somehow, the files size in the final index were 1MB, or 8MB - nothing in the
> middle. Am I missing something? Is the best way to really control the files
> size is to implement a custom Directory?
>
>
>
> Michael McCandless-2 wrote:
>>
>> So the rough upper bound on any segment's size is mergeFactor *
>> maxMergeMB.
>>
>
> --
> View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27092361.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
Well, this rule seems not working...
I tried to create an index of 90k documents, with different merge factors.
Somehow, the files size in the final index were 1MB, or 8MB - nothing in the
middle. Am I missing something? Is the best way to really control the files
size is to implement a custom Directory?
Michael McCandless-2 wrote:
>
> So the rough upper bound on any segment's size is mergeFactor *
> maxMergeMB.
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27092361.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Merge factor controls how many segments are merged at once. The default is 10.
The maxMergeMB setting sets the max size for a given segment to be
included in a merge.
Roughly, the upper bound on merged segments is the sum of their sizes.
So the rough upper bound on any segment's size is mergeFactor * maxMergeMB.
Mike
On Thu, Jan 7, 2010 at 11:04 AM, Dvora <ba...@gmail.com> wrote:
>
> Can you explain how the combination of merge factor and max merge size
> control the size of files?
>
> For example, if one would like to limit the files size to 3,4 or 7MB - how
> these parameters values can be predicted?
>
>
>
> Michael McCandless-2 wrote:
>>
>>
>> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
>> in size. With a default merge factor of 10, this should "generally"
>> mean you don't get a segment over 10MB, though it may not be a hard
>> guarantee (you can lower the 1.0 if you still see a segment over 10
>> MB).
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
Can you explain how the combination of merge factor and max merge size
control the size of files?
For example, if one would like to limit the files size to 3,4 or 7MB - how
these parameters values can be predicted?
Michael McCandless-2 wrote:
>
>
> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
> in size. With a default merge factor of 10, this should "generally"
> mean you don't get a segment over 10MB, though it may not be a hard
> guarantee (you can lower the 1.0 if you still see a segment over 10
> MB).
>
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27062291.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Jan 7, 2010 at 7:56 AM, Dvora <ba...@gmail.com> wrote:
> Yep, this seems to work :-) Thanks!
Super!
> The other way of Directory custom implementation seems interesting, and I
> would like try it (at least as an oppurtunity to know Lucene better) - is
> there a simple guide describing how to that? Perhaps some other similar
> Directory implementations?
I would start by looking @ FSDirectory/SimpleFSDirectory's source
code. They are simple wrappers around the corresponding java io APIs.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
Yep, this seems to work :-) Thanks!
The other way of Directory custom implementation seems interesting, and I
would like try it (at least as an oppurtunity to know Lucene better) - is
there a simple guide describing how to that? Perhaps some other similar
Directory implementations?
Michael McCandless-2 wrote:
>
> Is the creation of the index in your control? If so, try something like
> this:
>
> ((LogByteSizeMergePolicy) writer.getMergePolicy()).setMaxMergeMB(1.0));
>
> (NOTE: not tested). This works because [currently]
> LogByteSizeMergePolicy is the default for IndexWriter. Would be safer
> to create your own LogByteSizeMergePolicy and set it on the writer...
>
> This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
> in size. With a default merge factor of 10, this should "generally"
> mean you don't get a segment over 10MB, though it may not be a hard
> guarantee (you can lower the 1.0 if you still see a segment over 10
> MB).
>
> Implementing a custom directory that splits files @ 10 MB is also
> possible, but is a good amount more work. You'd have to have your
> IndexOutput write to the separate 10 MB sized files, then make an
> IndexInput that "pretends" they are one file.
>
> Mike
>
> On Thu, Jan 7, 2010 at 6:24 AM, Dvora <ba...@gmail.com> wrote:
>>
>> Thanks for the reply.
>>
>> Can you please add some detailed explanations? I'm trying to upload
>> Lucene
>> index to Google App Engine, and the files size must not exceed 10MB.
>>
>>
>>
>> Michael McCandless-2 wrote:
>>>
>>> I don't think this is implemented [yet] today. You'd have to
>>> implement the Directory, IndexInput and IndexOutput classes, to make
>>> this work.
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27058014.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27059555.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Is the creation of the index in your control? If so, try something like this:
((LogByteSizeMergePolicy) writer.getMergePolicy()).setMaxMergeMB(1.0));
(NOTE: not tested). This works because [currently]
LogByteSizeMergePolicy is the default for IndexWriter. Would be safer
to create your own LogByteSizeMergePolicy and set it on the writer...
This tells the IndexWriter NOT to merge any segment that's over 1.0 MB
in size. With a default merge factor of 10, this should "generally"
mean you don't get a segment over 10MB, though it may not be a hard
guarantee (you can lower the 1.0 if you still see a segment over 10
MB).
Implementing a custom directory that splits files @ 10 MB is also
possible, but is a good amount more work. You'd have to have your
IndexOutput write to the separate 10 MB sized files, then make an
IndexInput that "pretends" they are one file.
Mike
On Thu, Jan 7, 2010 at 6:24 AM, Dvora <ba...@gmail.com> wrote:
>
> Thanks for the reply.
>
> Can you please add some detailed explanations? I'm trying to upload Lucene
> index to Google App Engine, and the files size must not exceed 10MB.
>
>
>
> Michael McCandless-2 wrote:
>>
>> I don't think this is implemented [yet] today. You'd have to
>> implement the Directory, IndexInput and IndexOutput classes, to make
>> this work.
>>
>
> --
> View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27058014.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Dvora <ba...@gmail.com>.
Thanks for the reply.
Can you please add some detailed explanations? I'm trying to upload Lucene
index to Google App Engine, and the files size must not exceed 10MB.
Michael McCandless-2 wrote:
>
> I don't think this is implemented [yet] today. You'd have to
> implement the Directory, IndexInput and IndexOutput classes, to make
> this work.
>
--
View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27058014.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Is there a way to limit the size of an index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
I don't think this is implemented [yet] today. You'd have to
implement the Directory, IndexInput and IndexOutput classes, to make
this work.
Do you have a hard 2GB limit on index files? Setting
maxMergeDocs/Size (via LogByteSizeMergePolicy) lets you approximately
limit the size...
Mike
On Thu, Jan 7, 2010 at 3:32 AM, Dvora <ba...@gmail.com> wrote:
>
> Hi all,
>
> According to the FAQ, "An even more complex and optimal solution:
> Write a version of FSDirectory that, when a file exceeds 2GB, creates a
> subdirectory and represents the file as a series of files."
>
> Is that solution implemented already? If not, can you guide me please how
> can I achieve that? Is this solution requires changes in the Reader as well?
>
> Thanks.
>
>
> --
> View this message in context: http://old.nabble.com/Is-there-a-way-to-limit-the-size-of-an-index--tp27056573p27056573.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org