You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Hostetter <ho...@fucit.org> on 2010/04/05 19:59:30 UTC
Re: no of cfs files are more that the mergeFactor
This sounds completley normal form what i remembe about mergeFactor.
Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
5 "level 1" segments are formed they are merged into a single "level 2"
segment. then 5 more "level 1" segments are allowed to form before the
next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
sements, then they are all merged into a single "level 3" segment, etc...
: I had my mergeFactor as 5 ,
: but when i load a data with some 1,00,000 i got some 12 .cfs files in my
: data/index folder .
:
: How come this is possible .
: in what context we can have more no of .cfs files
-Hoss
Re: no of cfs files are more that the mergeFactor
Posted by Michael McCandless <lu...@mikemccandless.com>.
I like that name! That's a good way to think of it, assuming the
available coins/bill denominations grow exponentially with a base
roughly of mergeFactor :)
It's also like the odometer on a car.
Mike
On Tue, Apr 6, 2010 at 10:51 PM, Lance Norskog <go...@gmail.com> wrote:
> Ok, thanks. I'm studying the RAM buffer/MergePolicy nexus as we speak.
>
> I hereby name the function "minimum number of coins and bills needed
> to represent a number" as its "change log".
>
> On Tue, Apr 6, 2010 at 2:08 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> Actually this isn't quite right.
>>
>> Lucene flushes a new segment whenever RAM is full (not every 5 docs if
>> mergeFactor is 5).
>>
>> Whereas mergeFactor decides how many segments of roughly the same size
>> are merged at once.
>>
>> So eg if you index 42 docs, unless the docs are immense (or, are not
>> indexed in a single session), that will create 1 segment.
>>
>> Mike
>>
>> On Mon, Apr 5, 2010 at 6:21 PM, Lance Norskog <go...@gmail.com> wrote:
>>> mergeFactor=5 means that if there are 42 documents, there will be 3 index files:
>>>
>>> 1 with 25 documents,
>>> 3 with 5 documents, and
>>> 1 with 2 documents
>>>
>>> Imagine making change with coins of 1 document, 5 documents, 5^2
>>> documents, 5^3 documents, etc.
>>>
>>> On Mon, Apr 5, 2010 at 10:59 AM, Chris Hostetter
>>> <ho...@fucit.org> wrote:
>>>>
>>>> This sounds completley normal form what i remembe about mergeFactor.
>>>>
>>>> Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
>>>> 5 "level 1" segments are formed they are merged into a single "level 2"
>>>> segment. then 5 more "level 1" segments are allowed to form before the
>>>> next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
>>>> sements, then they are all merged into a single "level 3" segment, etc...
>>>>
>>>> : I had my mergeFactor as 5 ,
>>>> : but when i load a data with some 1,00,000 i got some 12 .cfs files in my
>>>> : data/index folder .
>>>> :
>>>> : How come this is possible .
>>>> : in what context we can have more no of .cfs files
>>>>
>>>>
>>>> -Hoss
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: no of cfs files are more that the mergeFactor
Posted by Lance Norskog <go...@gmail.com>.
Ok, thanks. I'm studying the RAM buffer/MergePolicy nexus as we speak.
I hereby name the function "minimum number of coins and bills needed
to represent a number" as its "change log".
On Tue, Apr 6, 2010 at 2:08 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Actually this isn't quite right.
>
> Lucene flushes a new segment whenever RAM is full (not every 5 docs if
> mergeFactor is 5).
>
> Whereas mergeFactor decides how many segments of roughly the same size
> are merged at once.
>
> So eg if you index 42 docs, unless the docs are immense (or, are not
> indexed in a single session), that will create 1 segment.
>
> Mike
>
> On Mon, Apr 5, 2010 at 6:21 PM, Lance Norskog <go...@gmail.com> wrote:
>> mergeFactor=5 means that if there are 42 documents, there will be 3 index files:
>>
>> 1 with 25 documents,
>> 3 with 5 documents, and
>> 1 with 2 documents
>>
>> Imagine making change with coins of 1 document, 5 documents, 5^2
>> documents, 5^3 documents, etc.
>>
>> On Mon, Apr 5, 2010 at 10:59 AM, Chris Hostetter
>> <ho...@fucit.org> wrote:
>>>
>>> This sounds completley normal form what i remembe about mergeFactor.
>>>
>>> Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
>>> 5 "level 1" segments are formed they are merged into a single "level 2"
>>> segment. then 5 more "level 1" segments are allowed to form before the
>>> next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
>>> sements, then they are all merged into a single "level 3" segment, etc...
>>>
>>> : I had my mergeFactor as 5 ,
>>> : but when i load a data with some 1,00,000 i got some 12 .cfs files in my
>>> : data/index folder .
>>> :
>>> : How come this is possible .
>>> : in what context we can have more no of .cfs files
>>>
>>>
>>> -Hoss
>>>
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
--
Lance Norskog
goksron@gmail.com
Re: no of cfs files are more that the mergeFactor
Posted by Michael McCandless <lu...@mikemccandless.com>.
Actually this isn't quite right.
Lucene flushes a new segment whenever RAM is full (not every 5 docs if
mergeFactor is 5).
Whereas mergeFactor decides how many segments of roughly the same size
are merged at once.
So eg if you index 42 docs, unless the docs are immense (or, are not
indexed in a single session), that will create 1 segment.
Mike
On Mon, Apr 5, 2010 at 6:21 PM, Lance Norskog <go...@gmail.com> wrote:
> mergeFactor=5 means that if there are 42 documents, there will be 3 index files:
>
> 1 with 25 documents,
> 3 with 5 documents, and
> 1 with 2 documents
>
> Imagine making change with coins of 1 document, 5 documents, 5^2
> documents, 5^3 documents, etc.
>
> On Mon, Apr 5, 2010 at 10:59 AM, Chris Hostetter
> <ho...@fucit.org> wrote:
>>
>> This sounds completley normal form what i remembe about mergeFactor.
>>
>> Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
>> 5 "level 1" segments are formed they are merged into a single "level 2"
>> segment. then 5 more "level 1" segments are allowed to form before the
>> next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
>> sements, then they are all merged into a single "level 3" segment, etc...
>>
>> : I had my mergeFactor as 5 ,
>> : but when i load a data with some 1,00,000 i got some 12 .cfs files in my
>> : data/index folder .
>> :
>> : How come this is possible .
>> : in what context we can have more no of .cfs files
>>
>>
>> -Hoss
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: no of cfs files are more that the mergeFactor
Posted by Lance Norskog <go...@gmail.com>.
mergeFactor=5 means that if there are 42 documents, there will be 3 index files:
1 with 25 documents,
3 with 5 documents, and
1 with 2 documents
Imagine making change with coins of 1 document, 5 documents, 5^2
documents, 5^3 documents, etc.
On Mon, Apr 5, 2010 at 10:59 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> This sounds completley normal form what i remembe about mergeFactor.
>
> Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
> 5 "level 1" segments are formed they are merged into a single "level 2"
> segment. then 5 more "level 1" segments are allowed to form before the
> next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
> sements, then they are all merged into a single "level 3" segment, etc...
>
> : I had my mergeFactor as 5 ,
> : but when i load a data with some 1,00,000 i got some 12 .cfs files in my
> : data/index folder .
> :
> : How come this is possible .
> : in what context we can have more no of .cfs files
>
>
> -Hoss
>
>
--
Lance Norskog
goksron@gmail.com
Re: no of cfs files are more that the mergeFactor
Posted by Mark Miller <ma...@gmail.com>.
I'm guessing the user is expecting there to be one cfs file for the
index, and does not understand that its actually per segment.
On 04/05/2010 01:59 PM, Chris Hostetter wrote:
> This sounds completley normal form what i remembe about mergeFactor.
>
> Segmenets are merged "by level" meaning that with a mergeFactor of 5, once
> 5 "level 1" segments are formed they are merged into a single "level 2"
> segment. then 5 more "level 1" segments are allowed to form before the
> next merge (resulting in 2 "legel 2" sements). Once you have 5 "level 2"
> sements, then they are all merged into a single "level 3" segment, etc...
>
> : I had my mergeFactor as 5 ,
> : but when i load a data with some 1,00,000 i got some 12 .cfs files in my
> : data/index folder .
> :
> : How come this is possible .
> : in what context we can have more no of .cfs files
>
>
> -Hoss
>
>
--
- Mark
http://www.lucidimagination.com