You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by googoo <li...@gmail.com> on 2014/05/06 21:39:46 UTC

solr optimize on fnm file

All,

We saw one issue on fnm file.
Looks fnm file size will not be reduced after optimize.

Like we have 1000 documents, and they have field 1 to 1000.
And .fnm file size is 10K.

After delete 999 documents and just keep one document which just has 2
fields.
After run optimize, .fnm file still has 1000 fields.
Expected result is .fnm file should just has 2 fields after optimize.


Thanks,
Yongtao





--
View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr optimize on fnm file

Posted by googoo <li...@gmail.com>.
Erick,

Thanks for your update.

The problem this this data is will being until whole document in the section
be deleted.
I understand this is cause optimize double scan index folder in this case.
We may add some logic to check when the file size do this scan when the file
size is too bigger. 

Yongtao



--
View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969p4136569.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr optimize on fnm file

Posted by Erick Erickson <er...@gmail.com>.
Oh my! I see what your problem is, but I rather doubt that it'll be
addressed. You've obviously been stress-testing the indexing process
and have a bunch of garbage left over that's not getting removed on
optimize. But it's such a unique case that I don't know if anyone is
really interested in fixing it.

I'll bring it up on the dev list, but I expect you'll need to blow
away your entire index and re-index from scratch.

Best,
Erick

On Tue, May 6, 2014 at 4:09 PM, googoo <li...@gmail.com> wrote:
> For our setup, the file size is 123M. Internal it has 2.6M fields.
>
> The problem is facet operation. It take a while for facet.
> we are stuck in below call stack for 11 second.
>
> java.util.HashMap.transfer(Unknown Source)
> java.util.HashMap.resize(Unknown Source)
> java.util.HashMap.addEntry(Unknown Source)
> java.util.HashMap.put(Unknown Source)
> org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:285)
> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:302)
> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:251)
> org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:276)
> org.apache.lucene.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:220)
> org.apache.lucene.search.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:1116)
> org.apache.lucene.search.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:1106)
> org.apache.solr.request.SimpleFacets.getFieldCacheCounts(SimpleFacets.java:574)
> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:429)
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:517)
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:252)
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:78)
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969p4135037.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr optimize on fnm file

Posted by googoo <li...@gmail.com>.
For our setup, the file size is 123M. Internal it has 2.6M fields.

The problem is facet operation. It take a while for facet.
we are stuck in below call stack for 11 second.

java.util.HashMap.transfer​(Unknown Source)
java.util.HashMap.resize​(Unknown Source)
java.util.HashMap.addEntry​(Unknown Source)
java.util.HashMap.put​(Unknown Source)
org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal​(FieldInfos.java:285)
org.apache.lucene.index.FieldInfos$Builder.add​(FieldInfos.java:302)
org.apache.lucene.index.FieldInfos$Builder.add​(FieldInfos.java:251)
org.apache.lucene.index.MultiFields.getMergedFieldInfos​(MultiFields.java:276)
org.apache.lucene.index.SlowCompositeReaderWrapper.getFieldInfos​(SlowCompositeReaderWrapper.java:220)
org.apache.lucene.search.FieldCacheImpl.getTermsIndex​(FieldCacheImpl.java:1116)
org.apache.lucene.search.FieldCacheImpl.getTermsIndex​(FieldCacheImpl.java:1106)
org.apache.solr.request.SimpleFacets.getFieldCacheCounts​(SimpleFacets.java:574)
org.apache.solr.request.SimpleFacets.getTermCounts​(SimpleFacets.java:429)
org.apache.solr.request.SimpleFacets.getFacetFieldCounts​(SimpleFacets.java:517)
org.apache.solr.request.SimpleFacets.getFacetCounts​(SimpleFacets.java:252)
org.apache.solr.handler.component.FacetComponent.process​(FacetComponent.java:78)



--
View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969p4135037.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr optimize on fnm file

Posted by Erick Erickson <er...@gmail.com>.
How big is the fnm file? While you may be technically correct, I'm not
sure it would be worth the effort, I rather expect this file to be
quite small.

Are you seeing a performance issue or is this more in the theoretical realm?

Best,
Erick

On Tue, May 6, 2014 at 1:23 PM, googoo <li...@gmail.com> wrote:
> I check implementation.
>
> in SegmentMerger.mergeFieldInfos
>
>   public void mergeFieldInfos() throws IOException {
>     for (AtomicReader reader : mergeState.readers) {
>       FieldInfos readerFieldInfos = reader.getFieldInfos();
>       for (FieldInfo fi : readerFieldInfos) {
>         fieldInfosBuilder.add(fi);
>       }
>     }
>     mergeState.fieldInfos = fieldInfosBuilder.finish();
>   }
>
> we do not check if the field still use in new documents.
> We just add all existing fields to new segments field.
>
> Will we fix this issue?
>
> Thanks,
> Yongtao
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969p4134972.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr optimize on fnm file

Posted by googoo <li...@gmail.com>.
I check implementation.

in SegmentMerger.mergeFieldInfos

  public void mergeFieldInfos() throws IOException {
    for (AtomicReader reader : mergeState.readers) {
      FieldInfos readerFieldInfos = reader.getFieldInfos();
      for (FieldInfo fi : readerFieldInfos) {
        fieldInfosBuilder.add(fi);
      }
    }
    mergeState.fieldInfos = fieldInfosBuilder.finish();
  }

we do not check if the field still use in new documents.
We just add all existing fields to new segments field.

Will we fix this issue?

Thanks,
Yongtao



--
View this message in context: http://lucene.472066.n3.nabble.com/solr-optimize-on-fnm-file-tp4134969p4134972.html
Sent from the Solr - User mailing list archive at Nabble.com.