You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Varun Thacker <va...@gmail.com> on 2014/04/29 14:03:53 UTC
No Compound Files
I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
merges take place on the index. However I was unsuccessful at it. What I am
doing wrong here.
Attaching a gist with -
1. Output when using NoMergePolicy.NO_COMPOUND_FILES
2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
3. The code snippet I used.
https://gist.github.com/vthacker/11398124
I tried it using Lucene 4.7
--
Regards,
Varun Thacker
http://www.vthacker.in/
Re: No Compound Files
Posted by Varun Thacker <va...@gmail.com>.
Created LUCENE-5633 for it.
On Tue, Apr 29, 2014 at 6:28 PM, Shai Erera <se...@gmail.com> wrote:
> NoMP means no merges, and indeed it seems silly that NoMP distinguishes
> between compound/non-compound settings. Perhaps it's rooted somewhere in
> the past, I don't remember.
>
> I checked and IndexWriter.addIndexes consults
> MP.useCompoundFile(segmentInfo) when it adds the segments. But maybe
> NoMP.useCompoundFile can be changed to return
> newSegment.info.isCompoundFile? I.e. it doesn't change the type of the new
> segment?
>
> Shai
>
>
> On Tue, Apr 29, 2014 at 3:50 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> > +1 to just have NoMergePolicy.INSTANCE
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Tue, Apr 29, 2014 at 8:07 AM, Robert Muir <rc...@gmail.com> wrote:
> > > I think NoMergePolicy.NO_COMPOUND_FILES and
> > > NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
> > > NoMergePolicy.INSTANCE
> > >
> > > If you want to change whether CFS is used by indexwriter flush, you
> > > need to set that in IndexWriterConfig.
> > >
> > > On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
> > > <va...@gmail.com> wrote:
> > >> I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
> > >> merges take place on the index. However I was unsuccessful at it. What
> > I am
> > >> doing wrong here.
> > >>
> > >> Attaching a gist with -
> > >> 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
> > >> 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
> > >> 3. The code snippet I used.
> > >>
> > >> https://gist.github.com/vthacker/11398124
> > >>
> > >> I tried it using Lucene 4.7
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >>
> > >> Regards,
> > >> Varun Thacker
> > >> http://www.vthacker.in/
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
--
Regards,
Varun Thacker
http://www.vthacker.in/
Re: No Compound Files
Posted by Shai Erera <se...@gmail.com>.
NoMP means no merges, and indeed it seems silly that NoMP distinguishes
between compound/non-compound settings. Perhaps it's rooted somewhere in
the past, I don't remember.
I checked and IndexWriter.addIndexes consults
MP.useCompoundFile(segmentInfo) when it adds the segments. But maybe
NoMP.useCompoundFile can be changed to return
newSegment.info.isCompoundFile? I.e. it doesn't change the type of the new
segment?
Shai
On Tue, Apr 29, 2014 at 3:50 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> +1 to just have NoMergePolicy.INSTANCE
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Apr 29, 2014 at 8:07 AM, Robert Muir <rc...@gmail.com> wrote:
> > I think NoMergePolicy.NO_COMPOUND_FILES and
> > NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
> > NoMergePolicy.INSTANCE
> >
> > If you want to change whether CFS is used by indexwriter flush, you
> > need to set that in IndexWriterConfig.
> >
> > On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
> > <va...@gmail.com> wrote:
> >> I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
> >> merges take place on the index. However I was unsuccessful at it. What
> I am
> >> doing wrong here.
> >>
> >> Attaching a gist with -
> >> 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
> >> 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
> >> 3. The code snippet I used.
> >>
> >> https://gist.github.com/vthacker/11398124
> >>
> >> I tried it using Lucene 4.7
> >>
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Regards,
> >> Varun Thacker
> >> http://www.vthacker.in/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: No Compound Files
Posted by Michael McCandless <lu...@mikemccandless.com>.
+1 to just have NoMergePolicy.INSTANCE
Mike McCandless
http://blog.mikemccandless.com
On Tue, Apr 29, 2014 at 8:07 AM, Robert Muir <rc...@gmail.com> wrote:
> I think NoMergePolicy.NO_COMPOUND_FILES and
> NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
> NoMergePolicy.INSTANCE
>
> If you want to change whether CFS is used by indexwriter flush, you
> need to set that in IndexWriterConfig.
>
> On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
> <va...@gmail.com> wrote:
>> I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
>> merges take place on the index. However I was unsuccessful at it. What I am
>> doing wrong here.
>>
>> Attaching a gist with -
>> 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
>> 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
>> 3. The code snippet I used.
>>
>> https://gist.github.com/vthacker/11398124
>>
>> I tried it using Lucene 4.7
>>
>>
>>
>>
>>
>> --
>>
>>
>> Regards,
>> Varun Thacker
>> http://www.vthacker.in/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: No Compound Files
Posted by Varun Thacker <va...@gmail.com>.
Thanks for the response. I was not aware of IWC.setUseCompoundFile .
@Shai this is what I feel is confusing - From what I understand
NoMergePolicy means no merges. Hence why have two separate options?
On Tue, Apr 29, 2014 at 5:44 PM, Shai Erera <se...@gmail.com> wrote:
> The problem is that compound files settings are split between MergePolicy
> and IndexWriterConfig. As documented on IWC.setUseCompoundFile, this
> setting controls how new segments are flushed, while the MP setting
> controls how merged segments are written.
>
> If we only offer NoMP.INSTANCE, what would it do w/ merged segments? always
> compound? always not-compound? But that won't solve the problem of new
> flushed segments, since that's controlled by IWC.
>
> If we can move all of that to IWC, I think it will remove the confusion..
> it always confuses me that I use NoMP.COMPUND, yet I see non-compound
> segments, until I remember to change the IWC setting.
>
> Shai
>
>
> On Tue, Apr 29, 2014 at 3:07 PM, Robert Muir <rc...@gmail.com> wrote:
>
> > I think NoMergePolicy.NO_COMPOUND_FILES and
> > NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
> > NoMergePolicy.INSTANCE
> >
> > If you want to change whether CFS is used by indexwriter flush, you
> > need to set that in IndexWriterConfig.
> >
> > On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
> > <va...@gmail.com> wrote:
> > > I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
> > > merges take place on the index. However I was unsuccessful at it. What
> I
> > am
> > > doing wrong here.
> > >
> > > Attaching a gist with -
> > > 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
> > > 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
> > > 3. The code snippet I used.
> > >
> > > https://gist.github.com/vthacker/11398124
> > >
> > > I tried it using Lucene 4.7
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Varun Thacker
> > > http://www.vthacker.in/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
--
Regards,
Varun Thacker
http://www.vthacker.in/
Re: No Compound Files
Posted by Robert Muir <rc...@gmail.com>.
On Tue, Apr 29, 2014 at 8:14 AM, Shai Erera <se...@gmail.com> wrote:
>
> If we only offer NoMP.INSTANCE, what would it do w/ merged segments? always
> compound? always not-compound?
it doesnt merge though.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: No Compound Files
Posted by Shai Erera <se...@gmail.com>.
The problem is that compound files settings are split between MergePolicy
and IndexWriterConfig. As documented on IWC.setUseCompoundFile, this
setting controls how new segments are flushed, while the MP setting
controls how merged segments are written.
If we only offer NoMP.INSTANCE, what would it do w/ merged segments? always
compound? always not-compound? But that won't solve the problem of new
flushed segments, since that's controlled by IWC.
If we can move all of that to IWC, I think it will remove the confusion..
it always confuses me that I use NoMP.COMPUND, yet I see non-compound
segments, until I remember to change the IWC setting.
Shai
On Tue, Apr 29, 2014 at 3:07 PM, Robert Muir <rc...@gmail.com> wrote:
> I think NoMergePolicy.NO_COMPOUND_FILES and
> NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
> NoMergePolicy.INSTANCE
>
> If you want to change whether CFS is used by indexwriter flush, you
> need to set that in IndexWriterConfig.
>
> On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
> <va...@gmail.com> wrote:
> > I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
> > merges take place on the index. However I was unsuccessful at it. What I
> am
> > doing wrong here.
> >
> > Attaching a gist with -
> > 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
> > 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
> > 3. The code snippet I used.
> >
> > https://gist.github.com/vthacker/11398124
> >
> > I tried it using Lucene 4.7
> >
> >
> >
> >
> >
> > --
> >
> >
> > Regards,
> > Varun Thacker
> > http://www.vthacker.in/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: No Compound Files
Posted by Robert Muir <rc...@gmail.com>.
I think NoMergePolicy.NO_COMPOUND_FILES and
NoMergePolicy.COMPOUND_FILES should be removed, and replaced with
NoMergePolicy.INSTANCE
If you want to change whether CFS is used by indexwriter flush, you
need to set that in IndexWriterConfig.
On Tue, Apr 29, 2014 at 8:03 AM, Varun Thacker
<va...@gmail.com> wrote:
> I wanted to use the NoMergePolicy.NO_COMPOUND_FILES to ensure that no
> merges take place on the index. However I was unsuccessful at it. What I am
> doing wrong here.
>
> Attaching a gist with -
> 1. Output when using NoMergePolicy.NO_COMPOUND_FILES
> 2. Output when using TieredMergePolicy with policy.setNoCFSRatio(0.0)
> 3. The code snippet I used.
>
> https://gist.github.com/vthacker/11398124
>
> I tried it using Lucene 4.7
>
>
>
>
>
> --
>
>
> Regards,
> Varun Thacker
> http://www.vthacker.in/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org