You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Peter Keegan <pe...@gmail.com> on 2010/02/22 17:43:24 UTC
IndexWriter.getReader.getVersion behavior
Using Lucene 2.9.1, I have the following pseudocode which gets repeated at
regular intervals:
1. FSDirectory dir = FSDirectory.open(java.io.File);
2. dir.setLockFactory(new SingleInstanceLockFactory());
3. IndexWriter writer = new IndexWriter(dir, Analyzer, false, maxFieldLen)
4. writer.getReader().getVersion();
5. writer.prepareCommit();
6. writer.getReader().getVersion();
7. writer.commit();
8. writer.close();
I'm using the version number to keep external data in synch with the index.
Usually, the version number from (6) is 1 greater than from (4) and the
version from (4) equals the version from the previous (6). At least once a
day, however, the version from (4) is 1 greater than from the previous (6).
What would explain this sporadic behavior of version numbers?
Thanks,
Peter
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
That should be fine!
Mike
On Fri, Feb 26, 2010 at 3:26 PM, Peter Keegan <pe...@gmail.com> wrote:
> Can IW.waitForMerges be called between 'prepareCommit' and 'commit'? That's
> when the app calls 'getReader' to create external data.
>
> Peter
>
> On Fri, Feb 26, 2010 at 3:15 PM, Peter Keegan <pe...@gmail.com>wrote:
>
>> Great, I'll give it a try.
>> Thanks!
>>
>>
>> On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>> Note that it's a BG merge (not commit)...
>>>
>>> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API?
>>> If you call that, then call .getReader().getVersion(), then close &
>>> open the writer, I think (but you better test to be sure!) the next
>>> .getReader().getVersion() should always match.
>>>
>>> Mike
>>>
>>> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <pe...@gmail.com>
>>> wrote:
>>> > Is there a way for the application to wait for the BG commit to finish
>>> > before it calls IW.close? If so, would this prevent the extra version?
>>> The
>>> > extra version causes the app. to think that the external data it
>>> committed
>>> > is out of synch with the index, which requires the app to do extra
>>> > processing to re-synch.
>>> >
>>> > Thanks,
>>> > Peter
>>> >
>>> >
>>> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
>>> > lucene@mikemccandless.com> wrote:
>>> >
>>> >> OK -- I can now see what happened.
>>> >>
>>> >> There was a merge still running, when you called IW.commit (Lucene
>>> >> Merge Thread #0). Because IW.commit does not wait for BG merges to
>>> >> finish, but IW.close does (by default), this means you'll pick up an
>>> >> extra version whenever a merge is running when you call close.
>>> >>
>>> >> Mike
>>> >>
>>> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com>
>>> >> wrote:
>>> >> > I'm pretty sure this output occurred when the version number skipped
>>> +1.
>>> >> > The line containing '============'. separates the close/open
>>> IndexWriter.
>>> >> >
>>> >> > IFD [Indexer]: setInfoStream
>>> >> >
>>> >>
>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
>>> >> > IW 9 [Indexer]: setInfoStream:
>>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>>> >> >
>>> >>
>>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
>>> >>
>>> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
>>> >> =16.0
>>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>>> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
>>> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16
>>> >> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
>>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>>> flushDocStores=false
>>> >> > numDocs=1456 numBufDelTerms=1456
>>> >> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
>>> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
>>> >> _h:C1291->_h
>>> >> > _i:C703->_h
>>> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j
>>> numDocs=1456
>>> >> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704
>>> >> newFlushedSize=4969789
>>> >> > docs/MB=307.202 new/old=29.966%
>>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>>> >> > isCommit = false]
>>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>>> >> > isCommit = false]
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1
>>> segments
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9
>>> segments
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b
>>> >> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
>>> >> > IW 9 [Indexer]: prepareCommit: flush
>>> >> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j
>>> >> docStoreOffset=1456
>>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
>>> >> > numBufDelTerms=509
>>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j
>>> >> > IW 9 [Indexer]: flush shared docStore segment _j
>>> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
>>> >> > numDocs=1965
>>> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
>>> >> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
>>> >> > docs/MB=287.727 new/old=24.788%
>>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>>> >> false]
>>> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
>>> >> docIDs
>>> >> > and 0 deleted queries on 11 segments.
>>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>>> >> false]
>>> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments
>>> >> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
>>> >> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
>>> >> > IW 9 [Indexer]: LMP: 1 to 11: add this merge
>>> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j [total 1 pending]
>>> >> > IW 9 [Indexer]: CMS: now merge
>>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j into _l [mergeDocStores]
>>> >> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
>>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
>>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j changeCount=7
>>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
>>> >> > IW 9 [Indexer]: now sync _k.fnm
>>> >> > IW 9 [Lucene Merge Thread #0]: now merge
>>> >> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>>> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
>>> >> > [mergeDocStores]
>>> >> > merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
>>> >> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
>>> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
>>> >> _k:C509->_j
>>> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j into _l [mergeDocStores]
>>> >> > IW 9 [Indexer]: now sync _k.tii
>>> >> > IW 9 [Indexer]: now sync _k.tis
>>> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
>>> >> > IW 9 [Indexer]: now sync _j.tis
>>> >> > IW 9 [Indexer]: now sync _j.tii
>>> >> > IW 9 [Indexer]: now sync _j.fdx
>>> >> > IW 9 [Indexer]: now sync _j.fdt
>>> >> > IW 9 [Indexer]: now sync _k.prx
>>> >> > IW 9 [Indexer]: now sync _k.nrm
>>> >> > IW 9 [Indexer]: now sync _j.nrm
>>> >> > IW 9 [Indexer]: now sync _j.prx
>>> >> > IW 9 [Indexer]: now sync _j.fnm
>>> >> > IW 9 [Indexer]: now sync _k.frq
>>> >> > IW 9 [Indexer]: now sync _j.frq
>>> >> > IW 9 [Indexer]: done all syncs
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k
>>> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]: flush shared docStore segment _k
>>> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k
>>> >> numDocs=0
>>> >> > IW 9 [Indexer]: commit: start
>>> >> > IW 9 [Indexer]: commit: already prepared
>>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
>>> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit =
>>> >> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
>>> >> > IFD [Indexer]: delete "segments_b"
>>> >> > IW 9 [Indexer]: commit: done
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]: now flush at close
>>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS: now merge
>>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
>>> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
>>> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>>> >> _g:C858->_f
>>> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
>>> >> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
>>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>>> ;
>>> >> > isCommit = false]
>>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>>> ;
>>> >> > isCommit = false]
>>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
>>> >> > IW 9 [Indexer]: now call final commit()
>>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
>>> >> > IW 9 [Indexer]: now sync _l.nrm
>>> >> > IW 9 [Indexer]: now sync _l.fnm
>>> >> > IW 9 [Indexer]: now sync _l.prx
>>> >> > IW 9 [Indexer]: now sync _l.frq
>>> >> > IW 9 [Indexer]: now sync _l.tis
>>> >> > IW 9 [Indexer]: now sync _l.fdx
>>> >> > IW 9 [Indexer]: now sync _l.fdt
>>> >> > IW 9 [Indexer]: now sync _l.tii
>>> >> > IW 9 [Indexer]: done all syncs
>>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit =
>>> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
>>> >> > IFD [Indexer]: delete "_h.prx"
>>> >> > IFD [Indexer]: delete "_f.tis"
>>> >> > IFD [Indexer]: delete "_g.tis"
>>> >> > IFD [Indexer]: delete "_d.tii"
>>> >> > IFD [Indexer]: delete "_e.frq"
>>> >> > IFD [Indexer]: delete "_k.fnm"
>>> >> > IFD [Indexer]: delete "_c.tii"
>>> >> > IFD [Indexer]: delete "_h.nrm"
>>> >> > IFD [Indexer]: delete "_i.tis"
>>> >> > IFD [Indexer]: delete "_k.tii"
>>> >> > IFD [Indexer]: delete "_h.fdt"
>>> >> > IFD [Indexer]: delete "_c.tis"
>>> >> > IFD [Indexer]: delete "_g.tii"
>>> >> > IFD [Indexer]: delete "_h.fdx"
>>> >> > IFD [Indexer]: delete "_c.nrm"
>>> >> > IFD [Indexer]: delete "_i.tii"
>>> >> > IFD [Indexer]: delete "_d.tis"
>>> >> > IFD [Indexer]: delete "_f.tii"
>>> >> > IFD [Indexer]: delete "_e.fnm"
>>> >> > IFD [Indexer]: delete "_k.tis"
>>> >> > IFD [Indexer]: delete "_d.frq"
>>> >> > IFD [Indexer]: delete "_f.nrm"
>>> >> > IFD [Indexer]: delete "_h.fnm"
>>> >> > IFD [Indexer]: delete "segments_c"
>>> >> > IFD [Indexer]: delete "_c.prx"
>>> >> > IFD [Indexer]: delete "_d.fnm"
>>> >> > IFD [Indexer]: delete "_h.frq"
>>> >> > IFD [Indexer]: delete "_h.tis"
>>> >> > IFD [Indexer]: delete "_f.fdt"
>>> >> > IFD [Indexer]: delete "_j.tis"
>>> >> > IFD [Indexer]: delete "_f.fdx"
>>> >> > IFD [Indexer]: delete "_g.nrm"
>>> >> > IFD [Indexer]: delete "_f.prx"
>>> >> > IFD [Indexer]: delete "_j.tii"
>>> >> > IFD [Indexer]: delete "_h.tii"
>>> >> > IFD [Indexer]: delete "_e.prx"
>>> >> > IFD [Indexer]: delete "_c.frq"
>>> >> > IFD [Indexer]: delete "_i.prx"
>>> >> > IFD [Indexer]: delete "_j.fdx"
>>> >> > IFD [Indexer]: delete "_i.nrm"
>>> >> > IFD [Indexer]: delete "_j.fdt"
>>> >> > IFD [Indexer]: delete "_k.prx"
>>> >> > IFD [Indexer]: delete "_b.nrm"
>>> >> > IFD [Indexer]: delete "_e.tis"
>>> >> > IFD [Indexer]: delete "_e.tii"
>>> >> > IFD [Indexer]: delete "_g.fnm"
>>> >> > IFD [Indexer]: delete "_k.nrm"
>>> >> > IFD [Indexer]: delete "_j.nrm"
>>> >> > IFD [Indexer]: delete "_j.prx"
>>> >> > IFD [Indexer]: delete "_e.nrm"
>>> >> > IFD [Indexer]: delete "_g.prx"
>>> >> > IFD [Indexer]: delete "_b.prx"
>>> >> > IFD [Indexer]: delete "_i.frq"
>>> >> > IFD [Indexer]: delete "_b.tis"
>>> >> > IFD [Indexer]: delete "_d.prx"
>>> >> > IFD [Indexer]: delete "_g.frq"
>>> >> > IFD [Indexer]: delete "_d.fdx"
>>> >> > IFD [Indexer]: delete "_b.tii"
>>> >> > IFD [Indexer]: delete "_b.frq"
>>> >> > IFD [Indexer]: delete "_d.fdt"
>>> >> > IFD [Indexer]: delete "_f.fnm"
>>> >> > IFD [Indexer]: delete "_j.fnm"
>>> >> > IFD [Indexer]: delete "_b.fnm"
>>> >> > IFD [Indexer]: delete "_d.nrm"
>>> >> > IFD [Indexer]: delete "_b.fdt"
>>> >> > IFD [Indexer]: delete "_k.frq"
>>> >> > IFD [Indexer]: delete "_c.fnm"
>>> >> > IFD [Indexer]: delete "_j.frq"
>>> >> > IFD [Indexer]: delete "_i.fnm"
>>> >> > IFD [Indexer]: delete "_f.frq"
>>> >> > IFD [Indexer]: delete "_b.fdx"
>>> >> > IW 9 [Indexer]: commit: done
>>> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
>>> >> >
>>> >> > =====================================================
>>> >> >
>>> >> > IFD [Indexer]: setInfoStream
>>> >> >
>>> >>
>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
>>> >> > IW 10 [Indexer]: setInfoStream:
>>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>>> >> >
>>> >>
>>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
>>> >>
>>> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
>>> >> =16.0
>>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>>> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
>>> >> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
>>> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16
>>> >> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
>>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>>> flushDocStores=false
>>> >> > numDocs=1526 numBufDelTerms=1526
>>> >> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
>>> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
>>> >> numDocs=1526
>>> >> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
>>> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
>>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>>> >> > isCommit = false]
>>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>>> >> > isCommit = false]
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2
>>> >> segments
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1
>>> >> segments
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909
>>> _m:C1526->_m
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now
>>> return
>>> >> > IW 10 [Indexer]: prepareCommit: flush
>>> >> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m
>>> >> docStoreOffset=1526
>>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
>>> >> > numBufDelTerms=501
>>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > IW 10 [Indexer]: flush shared docStore segment _m
>>> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
>>> >> > numDocs=2027
>>> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
>>> >> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
>>> >> > docs/MB=316.593 new/old=23.953%
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>>> >> false]
>>> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
>>> >> docIDs
>>> >> > and 0 deleted queries on 4 segments.
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>>> >> false]
>>> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments
>>> >> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
>>> >> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
>>> >> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
>>> >> > IW 10 [Indexer]: CMS: now merge
>>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
>>> _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
>>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m changeCount=6
>>> >> > IW 10 [Indexer]: now sync _m.frq
>>> >> > IW 10 [Indexer]: now sync _m.prx
>>> >> > IW 10 [Indexer]: now sync _n.tis
>>> >> > IW 10 [Indexer]: now sync _m.tis
>>> >> > IW 10 [Indexer]: now sync _n.frq
>>> >> > IW 10 [Indexer]: now sync _n.tii
>>> >> > IW 10 [Indexer]: now sync _m.tii
>>> >> > IW 10 [Indexer]: now sync _m.fnm
>>> >> > IW 10 [Indexer]: now sync _n.fnm
>>> >> > IW 10 [Indexer]: now sync _n.nrm
>>> >> > IW 10 [Indexer]: now sync _n.prx
>>> >> > IW 10 [Indexer]: now sync _m.nrm
>>> >> > IW 10 [Indexer]: now sync _m.fdt
>>> >> > IW 10 [Indexer]: now sync _m.fdx
>>> >> > IW 10 [Indexer]: done all syncs
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]: flush shared docStore segment _n
>>> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n
>>> >> numDocs=0
>>> >> > IW 10 [Indexer]: commit: start
>>> >> > IW 10 [Indexer]: commit: already prepared
>>> >> > IW 10 [Indexer]: commit: pendingCommit != null
>>> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
>>> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit =
>>> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
>>> >> > IFD [Indexer]: delete "segments_d"
>>> >> > IW 10 [Indexer]: commit: done
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]: now flush at close
>>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS: now merge
>>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
>>> _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
>>> >> > IW 10 [Indexer]: now call final commit()
>>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 10 [Indexer]: skip startCommit(): no changes pending
>>> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip
>>> >> > IW 10 [Indexer]: commit: done
>>> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>>> >> >
>>> >> >
>>> >> > Peter
>>> >> >
>>> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
>>> >> > lucene@mikemccandless.com> wrote:
>>> >> >
>>> >> >> Do you know the place in the infoStream output where you got a
>>> reader
>>> >> >> with the wrong (unexplained extra +1) version? If so, can you post
>>> >> >> the infoStream output up to that point?
>>> >> >>
>>> >> >> Mike
>>> >> >>
>>> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <
>>> peterlkeegan@gmail.com>
>>> >> >> wrote:
>>> >> >> > I've reproduced this and I have a bunch of infoStream log files.
>>> Since
>>> >> >> the
>>> >> >> > messages have no timestamps, it's hard to tell where the relevant
>>> >> entries
>>> >> >> > are. What should I be looking for?
>>> >> >> >
>>> >> >> > Peter
>>> >> >> >
>>> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <
>>> peterlkeegan@gmail.com
>>> >> >> >wrote:
>>> >> >> >
>>> >> >> >> I'm pretty sure there are flushes and segment merges going on,
>>> but as
>>> >> >> you
>>> >> >> >> said, that shouldn't affect the version increment. I'll see what
>>> I
>>> >> can
>>> >> >> do to
>>> >> >> >> get infoStream output.
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> Peter
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
>>> >> >> >> lucene@mikemccandless.com> wrote:
>>> >> >> >>
>>> >> >> >>> Well I'm at a loss then. The version should only increment on
>>> >> commit.
>>> >> >> >>>
>>> >> >> >>> Can you make it all happen when infoStream is on, and post back?
>>> >> >> >>>
>>> >> >> >>> Mike
>>> >> >> >>>
>>> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <
>>> >> peterlkeegan@gmail.com
>>> >> >> >
>>> >> >> >>> wrote:
>>> >> >> >>> > Only one writer thread and one writer process.
>>> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean
>>> create,
>>> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
>>> >> >> >>> >
>>> >> >> >>> > Peter
>>> >> >> >>> >
>>> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>>> >> >> >>> > lucene@mikemccandless.com> wrote:
>>> >> >> >>> >
>>> >> >> >>> >> That's curious.
>>> >> >> >>> >>
>>> >> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first
>>> >> prepare,
>>> >> >> >>> >> since that will call prepareCommit internally) that this
>>> version
>>> >> >> >>> >> should increase.
>>> >> >> >>> >>
>>> >> >> >>> >> Is there only 1 thread doing this?
>>> >> >> >>> >>
>>> >> >> >>> >> Oh, and, are you passing false for autoCommit?
>>> >> >> >>> >>
>>> >> >> >>> >> Mike
>>> >> >> >>> >>
>>> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
>>> >> >> peterlkeegan@gmail.com
>>> >> >> >>> >
>>> >> >> >>> >> wrote:
>>> >> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which
>>> gets
>>> >> >> >>> repeated
>>> >> >> >>> >> at
>>> >> >> >>> >> > regular intervals:
>>> >> >> >>> >> >
>>> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>>> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>>> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer,
>>> false,
>>> >> >> >>> >> maxFieldLen)
>>> >> >> >>> >> > 4. writer.getReader().getVersion();
>>> >> >> >>> >> > 5. writer.prepareCommit();
>>> >> >> >>> >> > 6. writer.getReader().getVersion();
>>> >> >> >>> >> > 7. writer.commit();
>>> >> >> >>> >> > 8. writer.close();
>>> >> >> >>> >> >
>>> >> >> >>> >> > I'm using the version number to keep external data in synch
>>> >> with
>>> >> >> the
>>> >> >> >>> >> index.
>>> >> >> >>> >> > Usually, the version number from (6) is 1 greater than from
>>> (4)
>>> >> >> and
>>> >> >> >>> the
>>> >> >> >>> >> > version from (4) equals the version from the previous (6).
>>> At
>>> >> >> least
>>> >> >> >>> once
>>> >> >> >>> >> a
>>> >> >> >>> >> > day, however, the version from (4) is 1 greater than from
>>> the
>>> >> >> >>> previous
>>> >> >> >>> >> (6).
>>> >> >> >>> >> > What would explain this sporadic behavior of version
>>> numbers?
>>> >> >> >>> >> >
>>> >> >> >>> >> > Thanks,
>>> >> >> >>> >> > Peter
>>> >> >> >>> >> >
>>> >> >> >>> >>
>>> >> >> >>> >>
>>> >> >>
>>> ---------------------------------------------------------------------
>>> >> >> >>> >> To unsubscribe, e-mail:
>>> java-user-unsubscribe@lucene.apache.org
>>> >> >> >>> >> For additional commands, e-mail:
>>> >> java-user-help@lucene.apache.org
>>> >> >> >>> >>
>>> >> >> >>> >>
>>> >> >> >>> >
>>> >> >> >>>
>>> >> >> >>>
>>> >> ---------------------------------------------------------------------
>>> >> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> >> >>> For additional commands, e-mail:
>>> java-user-help@lucene.apache.org
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>
>>> >> >> >
>>> >> >>
>>> >> >>
>>> ---------------------------------------------------------------------
>>> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
Can IW.waitForMerges be called between 'prepareCommit' and 'commit'? That's
when the app calls 'getReader' to create external data.
Peter
On Fri, Feb 26, 2010 at 3:15 PM, Peter Keegan <pe...@gmail.com>wrote:
> Great, I'll give it a try.
> Thanks!
>
>
> On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Note that it's a BG merge (not commit)...
>>
>> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API?
>> If you call that, then call .getReader().getVersion(), then close &
>> open the writer, I think (but you better test to be sure!) the next
>> .getReader().getVersion() should always match.
>>
>> Mike
>>
>> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <pe...@gmail.com>
>> wrote:
>> > Is there a way for the application to wait for the BG commit to finish
>> > before it calls IW.close? If so, would this prevent the extra version?
>> The
>> > extra version causes the app. to think that the external data it
>> committed
>> > is out of synch with the index, which requires the app to do extra
>> > processing to re-synch.
>> >
>> > Thanks,
>> > Peter
>> >
>> >
>> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
>> > lucene@mikemccandless.com> wrote:
>> >
>> >> OK -- I can now see what happened.
>> >>
>> >> There was a merge still running, when you called IW.commit (Lucene
>> >> Merge Thread #0). Because IW.commit does not wait for BG merges to
>> >> finish, but IW.close does (by default), this means you'll pick up an
>> >> extra version whenever a merge is running when you call close.
>> >>
>> >> Mike
>> >>
>> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com>
>> >> wrote:
>> >> > I'm pretty sure this output occurred when the version number skipped
>> +1.
>> >> > The line containing '============'. separates the close/open
>> IndexWriter.
>> >> >
>> >> > IFD [Indexer]: setInfoStream
>> >> >
>> >>
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
>> >> > IW 9 [Indexer]: setInfoStream:
>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>> >> >
>> >>
>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
>> >>
>> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
>> >> =16.0
>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
>> >> _d:C1220->_d
>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> >> > IW 9 [Indexer]: flush at getReader
>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
>> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16
>> >> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>> flushDocStores=false
>> >> > numDocs=1456 numBufDelTerms=1456
>> >> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
>> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
>> >> _h:C1291->_h
>> >> > _i:C703->_h
>> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j
>> numDocs=1456
>> >> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704
>> >> newFlushedSize=4969789
>> >> > docs/MB=307.202 new/old=29.966%
>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>> >> > isCommit = false]
>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>> >> > isCommit = false]
>> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1
>> segments
>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9
>> segments
>> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
>> >> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b
>> >> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j
>> >> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
>> >> > IW 9 [Indexer]: prepareCommit: flush
>> >> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j
>> >> docStoreOffset=1456
>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
>> >> > numBufDelTerms=509
>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j
>> >> > IW 9 [Indexer]: flush shared docStore segment _j
>> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
>> >> > numDocs=1965
>> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
>> >> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
>> >> > docs/MB=287.727 new/old=24.788%
>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>> >> false]
>> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
>> >> docIDs
>> >> > and 0 deleted queries on 11 segments.
>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>> >> false]
>> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments
>> >> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
>> >> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
>> >> > IW 9 [Indexer]: LMP: 1 to 11: add this merge
>> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j [total 1 pending]
>> >> > IW 9 [Indexer]: CMS: now merge
>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>> >> _d:C1220->_d
>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> >> _j:C1456->_j
>> >> > _k:C509->_j
>> >> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b
>> >> _d:C1220->_d
>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> >> _j:C1456->_j
>> >> > _k:C509->_j into _l [mergeDocStores]
>> >> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j changeCount=7
>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
>> >> > IW 9 [Indexer]: now sync _k.fnm
>> >> > IW 9 [Lucene Merge Thread #0]: now merge
>> >> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
>> >> > [mergeDocStores]
>> >> > merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
>> >> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
>> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
>> >> _k:C509->_j
>> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
>> >> _d:C1220->_d
>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> >> _j:C1456->_j
>> >> > _k:C509->_j into _l [mergeDocStores]
>> >> > IW 9 [Indexer]: now sync _k.tii
>> >> > IW 9 [Indexer]: now sync _k.tis
>> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
>> >> > IW 9 [Indexer]: now sync _j.tis
>> >> > IW 9 [Indexer]: now sync _j.tii
>> >> > IW 9 [Indexer]: now sync _j.fdx
>> >> > IW 9 [Indexer]: now sync _j.fdt
>> >> > IW 9 [Indexer]: now sync _k.prx
>> >> > IW 9 [Indexer]: now sync _k.nrm
>> >> > IW 9 [Indexer]: now sync _j.nrm
>> >> > IW 9 [Indexer]: now sync _j.prx
>> >> > IW 9 [Indexer]: now sync _j.fnm
>> >> > IW 9 [Indexer]: now sync _k.frq
>> >> > IW 9 [Indexer]: now sync _j.frq
>> >> > IW 9 [Indexer]: done all syncs
>> >> > IW 9 [Indexer]: flush at getReader
>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k
>> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j
>> >> > IW 9 [Indexer]: flush shared docStore segment _k
>> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k
>> >> numDocs=0
>> >> > IW 9 [Indexer]: commit: start
>> >> > IW 9 [Indexer]: commit: already prepared
>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
>> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit =
>> >> true]
>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
>> >> > IFD [Indexer]: delete "segments_b"
>> >> > IW 9 [Indexer]: commit: done
>> >> > IW 9 [Indexer]: flush at getReader
>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j
>> >> > IW 9 [Indexer]: now flush at close
>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j
>> >> > IW 9 [Indexer]: CMS: now merge
>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>> >> _d:C1220->_d
>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> >> _j:C1456->_j
>> >> > _k:C509->_j
>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
>> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
>> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>> >> _g:C858->_f
>> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
>> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
>> >> _c:C717->_b
>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> >> _i:C703->_h
>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>> ;
>> >> > isCommit = false]
>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>> ;
>> >> > isCommit = false]
>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
>> >> > IW 9 [Indexer]: now call final commit()
>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
>> >> > IW 9 [Indexer]: now sync _l.nrm
>> >> > IW 9 [Indexer]: now sync _l.fnm
>> >> > IW 9 [Indexer]: now sync _l.prx
>> >> > IW 9 [Indexer]: now sync _l.frq
>> >> > IW 9 [Indexer]: now sync _l.tis
>> >> > IW 9 [Indexer]: now sync _l.fdx
>> >> > IW 9 [Indexer]: now sync _l.fdt
>> >> > IW 9 [Indexer]: now sync _l.tii
>> >> > IW 9 [Indexer]: done all syncs
>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
>> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit =
>> true]
>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
>> >> > IFD [Indexer]: delete "_h.prx"
>> >> > IFD [Indexer]: delete "_f.tis"
>> >> > IFD [Indexer]: delete "_g.tis"
>> >> > IFD [Indexer]: delete "_d.tii"
>> >> > IFD [Indexer]: delete "_e.frq"
>> >> > IFD [Indexer]: delete "_k.fnm"
>> >> > IFD [Indexer]: delete "_c.tii"
>> >> > IFD [Indexer]: delete "_h.nrm"
>> >> > IFD [Indexer]: delete "_i.tis"
>> >> > IFD [Indexer]: delete "_k.tii"
>> >> > IFD [Indexer]: delete "_h.fdt"
>> >> > IFD [Indexer]: delete "_c.tis"
>> >> > IFD [Indexer]: delete "_g.tii"
>> >> > IFD [Indexer]: delete "_h.fdx"
>> >> > IFD [Indexer]: delete "_c.nrm"
>> >> > IFD [Indexer]: delete "_i.tii"
>> >> > IFD [Indexer]: delete "_d.tis"
>> >> > IFD [Indexer]: delete "_f.tii"
>> >> > IFD [Indexer]: delete "_e.fnm"
>> >> > IFD [Indexer]: delete "_k.tis"
>> >> > IFD [Indexer]: delete "_d.frq"
>> >> > IFD [Indexer]: delete "_f.nrm"
>> >> > IFD [Indexer]: delete "_h.fnm"
>> >> > IFD [Indexer]: delete "segments_c"
>> >> > IFD [Indexer]: delete "_c.prx"
>> >> > IFD [Indexer]: delete "_d.fnm"
>> >> > IFD [Indexer]: delete "_h.frq"
>> >> > IFD [Indexer]: delete "_h.tis"
>> >> > IFD [Indexer]: delete "_f.fdt"
>> >> > IFD [Indexer]: delete "_j.tis"
>> >> > IFD [Indexer]: delete "_f.fdx"
>> >> > IFD [Indexer]: delete "_g.nrm"
>> >> > IFD [Indexer]: delete "_f.prx"
>> >> > IFD [Indexer]: delete "_j.tii"
>> >> > IFD [Indexer]: delete "_h.tii"
>> >> > IFD [Indexer]: delete "_e.prx"
>> >> > IFD [Indexer]: delete "_c.frq"
>> >> > IFD [Indexer]: delete "_i.prx"
>> >> > IFD [Indexer]: delete "_j.fdx"
>> >> > IFD [Indexer]: delete "_i.nrm"
>> >> > IFD [Indexer]: delete "_j.fdt"
>> >> > IFD [Indexer]: delete "_k.prx"
>> >> > IFD [Indexer]: delete "_b.nrm"
>> >> > IFD [Indexer]: delete "_e.tis"
>> >> > IFD [Indexer]: delete "_e.tii"
>> >> > IFD [Indexer]: delete "_g.fnm"
>> >> > IFD [Indexer]: delete "_k.nrm"
>> >> > IFD [Indexer]: delete "_j.nrm"
>> >> > IFD [Indexer]: delete "_j.prx"
>> >> > IFD [Indexer]: delete "_e.nrm"
>> >> > IFD [Indexer]: delete "_g.prx"
>> >> > IFD [Indexer]: delete "_b.prx"
>> >> > IFD [Indexer]: delete "_i.frq"
>> >> > IFD [Indexer]: delete "_b.tis"
>> >> > IFD [Indexer]: delete "_d.prx"
>> >> > IFD [Indexer]: delete "_g.frq"
>> >> > IFD [Indexer]: delete "_d.fdx"
>> >> > IFD [Indexer]: delete "_b.tii"
>> >> > IFD [Indexer]: delete "_b.frq"
>> >> > IFD [Indexer]: delete "_d.fdt"
>> >> > IFD [Indexer]: delete "_f.fnm"
>> >> > IFD [Indexer]: delete "_j.fnm"
>> >> > IFD [Indexer]: delete "_b.fnm"
>> >> > IFD [Indexer]: delete "_d.nrm"
>> >> > IFD [Indexer]: delete "_b.fdt"
>> >> > IFD [Indexer]: delete "_k.frq"
>> >> > IFD [Indexer]: delete "_c.fnm"
>> >> > IFD [Indexer]: delete "_j.frq"
>> >> > IFD [Indexer]: delete "_i.fnm"
>> >> > IFD [Indexer]: delete "_f.frq"
>> >> > IFD [Indexer]: delete "_b.fdx"
>> >> > IW 9 [Indexer]: commit: done
>> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
>> >> >
>> >> > =====================================================
>> >> >
>> >> > IFD [Indexer]: setInfoStream
>> >> >
>> >>
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
>> >> > IW 10 [Indexer]: setInfoStream:
>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>> >> >
>> >>
>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
>> >>
>> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
>> >> =16.0
>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909
>> >> > IW 10 [Indexer]: flush at getReader
>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
>> >> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
>> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16
>> >> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>> flushDocStores=false
>> >> > numDocs=1526 numBufDelTerms=1526
>> >> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
>> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
>> >> numDocs=1526
>> >> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
>> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>> >> > isCommit = false]
>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>> >> > isCommit = false]
>> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2
>> >> segments
>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1
>> >> segments
>> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
>> >> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909
>> _m:C1526->_m
>> >> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now
>> return
>> >> > IW 10 [Indexer]: prepareCommit: flush
>> >> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m
>> >> docStoreOffset=1526
>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
>> >> > numBufDelTerms=501
>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> >> > IW 10 [Indexer]: flush shared docStore segment _m
>> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
>> >> > numDocs=2027
>> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
>> >> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
>> >> > docs/MB=316.593 new/old=23.953%
>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>> >> false]
>> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
>> >> docIDs
>> >> > and 0 deleted queries on 4 segments.
>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>> >> false]
>> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments
>> >> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
>> >> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
>> >> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
>> >> > IW 10 [Indexer]: CMS: now merge
>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
>> _n:C501->_m
>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
>> >> > _n:C501->_m changeCount=6
>> >> > IW 10 [Indexer]: now sync _m.frq
>> >> > IW 10 [Indexer]: now sync _m.prx
>> >> > IW 10 [Indexer]: now sync _n.tis
>> >> > IW 10 [Indexer]: now sync _m.tis
>> >> > IW 10 [Indexer]: now sync _n.frq
>> >> > IW 10 [Indexer]: now sync _n.tii
>> >> > IW 10 [Indexer]: now sync _m.tii
>> >> > IW 10 [Indexer]: now sync _m.fnm
>> >> > IW 10 [Indexer]: now sync _n.fnm
>> >> > IW 10 [Indexer]: now sync _n.nrm
>> >> > IW 10 [Indexer]: now sync _n.prx
>> >> > IW 10 [Indexer]: now sync _m.nrm
>> >> > IW 10 [Indexer]: now sync _m.fdt
>> >> > IW 10 [Indexer]: now sync _m.fdx
>> >> > IW 10 [Indexer]: done all syncs
>> >> > IW 10 [Indexer]: flush at getReader
>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> >> > _n:C501->_m
>> >> > IW 10 [Indexer]: flush shared docStore segment _n
>> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n
>> >> numDocs=0
>> >> > IW 10 [Indexer]: commit: start
>> >> > IW 10 [Indexer]: commit: already prepared
>> >> > IW 10 [Indexer]: commit: pendingCommit != null
>> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
>> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit =
>> true]
>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
>> >> > IFD [Indexer]: delete "segments_d"
>> >> > IW 10 [Indexer]: commit: done
>> >> > IW 10 [Indexer]: flush at getReader
>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> >> > _n:C501->_m
>> >> > IW 10 [Indexer]: now flush at close
>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> >> > numBufDelTerms=0
>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> >> > _n:C501->_m
>> >> > IW 10 [Indexer]: CMS: now merge
>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
>> _n:C501->_m
>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
>> >> > IW 10 [Indexer]: now call final commit()
>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>> >> > IW 10 [Indexer]: skip startCommit(): no changes pending
>> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip
>> >> > IW 10 [Indexer]: commit: done
>> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>> >> >
>> >> >
>> >> > Peter
>> >> >
>> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
>> >> > lucene@mikemccandless.com> wrote:
>> >> >
>> >> >> Do you know the place in the infoStream output where you got a
>> reader
>> >> >> with the wrong (unexplained extra +1) version? If so, can you post
>> >> >> the infoStream output up to that point?
>> >> >>
>> >> >> Mike
>> >> >>
>> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <
>> peterlkeegan@gmail.com>
>> >> >> wrote:
>> >> >> > I've reproduced this and I have a bunch of infoStream log files.
>> Since
>> >> >> the
>> >> >> > messages have no timestamps, it's hard to tell where the relevant
>> >> entries
>> >> >> > are. What should I be looking for?
>> >> >> >
>> >> >> > Peter
>> >> >> >
>> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <
>> peterlkeegan@gmail.com
>> >> >> >wrote:
>> >> >> >
>> >> >> >> I'm pretty sure there are flushes and segment merges going on,
>> but as
>> >> >> you
>> >> >> >> said, that shouldn't affect the version increment. I'll see what
>> I
>> >> can
>> >> >> do to
>> >> >> >> get infoStream output.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Peter
>> >> >> >>
>> >> >> >>
>> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
>> >> >> >> lucene@mikemccandless.com> wrote:
>> >> >> >>
>> >> >> >>> Well I'm at a loss then. The version should only increment on
>> >> commit.
>> >> >> >>>
>> >> >> >>> Can you make it all happen when infoStream is on, and post back?
>> >> >> >>>
>> >> >> >>> Mike
>> >> >> >>>
>> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <
>> >> peterlkeegan@gmail.com
>> >> >> >
>> >> >> >>> wrote:
>> >> >> >>> > Only one writer thread and one writer process.
>> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean
>> create,
>> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
>> >> >> >>> >
>> >> >> >>> > Peter
>> >> >> >>> >
>> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>> >> >> >>> > lucene@mikemccandless.com> wrote:
>> >> >> >>> >
>> >> >> >>> >> That's curious.
>> >> >> >>> >>
>> >> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first
>> >> prepare,
>> >> >> >>> >> since that will call prepareCommit internally) that this
>> version
>> >> >> >>> >> should increase.
>> >> >> >>> >>
>> >> >> >>> >> Is there only 1 thread doing this?
>> >> >> >>> >>
>> >> >> >>> >> Oh, and, are you passing false for autoCommit?
>> >> >> >>> >>
>> >> >> >>> >> Mike
>> >> >> >>> >>
>> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
>> >> >> peterlkeegan@gmail.com
>> >> >> >>> >
>> >> >> >>> >> wrote:
>> >> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which
>> gets
>> >> >> >>> repeated
>> >> >> >>> >> at
>> >> >> >>> >> > regular intervals:
>> >> >> >>> >> >
>> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer,
>> false,
>> >> >> >>> >> maxFieldLen)
>> >> >> >>> >> > 4. writer.getReader().getVersion();
>> >> >> >>> >> > 5. writer.prepareCommit();
>> >> >> >>> >> > 6. writer.getReader().getVersion();
>> >> >> >>> >> > 7. writer.commit();
>> >> >> >>> >> > 8. writer.close();
>> >> >> >>> >> >
>> >> >> >>> >> > I'm using the version number to keep external data in synch
>> >> with
>> >> >> the
>> >> >> >>> >> index.
>> >> >> >>> >> > Usually, the version number from (6) is 1 greater than from
>> (4)
>> >> >> and
>> >> >> >>> the
>> >> >> >>> >> > version from (4) equals the version from the previous (6).
>> At
>> >> >> least
>> >> >> >>> once
>> >> >> >>> >> a
>> >> >> >>> >> > day, however, the version from (4) is 1 greater than from
>> the
>> >> >> >>> previous
>> >> >> >>> >> (6).
>> >> >> >>> >> > What would explain this sporadic behavior of version
>> numbers?
>> >> >> >>> >> >
>> >> >> >>> >> > Thanks,
>> >> >> >>> >> > Peter
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> >>> >> To unsubscribe, e-mail:
>> java-user-unsubscribe@lucene.apache.org
>> >> >> >>> >> For additional commands, e-mail:
>> >> java-user-help@lucene.apache.org
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >
>> >> >> >>>
>> >> >> >>>
>> >> ---------------------------------------------------------------------
>> >> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> >> >>> For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>> >> >> >>>
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> >>
>> >> >>
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
Great, I'll give it a try.
Thanks!
On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Note that it's a BG merge (not commit)...
>
> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API?
> If you call that, then call .getReader().getVersion(), then close &
> open the writer, I think (but you better test to be sure!) the next
> .getReader().getVersion() should always match.
>
> Mike
>
> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <pe...@gmail.com>
> wrote:
> > Is there a way for the application to wait for the BG commit to finish
> > before it calls IW.close? If so, would this prevent the extra version?
> The
> > extra version causes the app. to think that the external data it
> committed
> > is out of synch with the index, which requires the app to do extra
> > processing to re-synch.
> >
> > Thanks,
> > Peter
> >
> >
> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> OK -- I can now see what happened.
> >>
> >> There was a merge still running, when you called IW.commit (Lucene
> >> Merge Thread #0). Because IW.commit does not wait for BG merges to
> >> finish, but IW.close does (by default), this means you'll pick up an
> >> extra version whenever a merge is running when you call close.
> >>
> >> Mike
> >>
> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com>
> >> wrote:
> >> > I'm pretty sure this output occurred when the version number skipped
> +1.
> >> > The line containing '============'. separates the close/open
> IndexWriter.
> >> >
> >> > IFD [Indexer]: setInfoStream
> >> >
> >>
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
> >> > IW 9 [Indexer]: setInfoStream:
> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> >> > pathname>\lresumes1.search.main.1 autoCommit=false
> >> >
> >>
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
> >>
> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
> >> =16.0
> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
> >> _d:C1220->_d
> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> >> > IW 9 [Indexer]: flush at getReader
> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16
> >> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
> flushDocStores=false
> >> > numDocs=1456 numBufDelTerms=1456
> >> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
> >> _h:C1291->_h
> >> > _i:C703->_h
> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j
> numDocs=1456
> >> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704
> >> newFlushedSize=4969789
> >> > docs/MB=307.202 new/old=29.966%
> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> >> > isCommit = false]
> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> >> > isCommit = false]
> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1
> segments
> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9
> segments
> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
> >> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b
> >> _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j
> >> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
> >> > IW 9 [Indexer]: prepareCommit: flush
> >> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j
> >> docStoreOffset=1456
> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
> >> > numBufDelTerms=509
> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j
> >> > IW 9 [Indexer]: flush shared docStore segment _j
> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
> >> > numDocs=1965
> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
> >> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
> >> > docs/MB=287.727 new/old=24.788%
> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
> >> false]
> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
> >> docIDs
> >> > and 0 deleted queries on 11 segments.
> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
> >> false]
> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments
> >> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
> >> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
> >> > IW 9 [Indexer]: LMP: 1 to 11: add this merge
> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j [total 1 pending]
> >> > IW 9 [Indexer]: CMS: now merge
> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
> >> _d:C1220->_d
> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> >> _j:C1456->_j
> >> > _k:C509->_j
> >> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b
> >> _d:C1220->_d
> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> >> _j:C1456->_j
> >> > _k:C509->_j into _l [mergeDocStores]
> >> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j changeCount=7
> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
> >> > IW 9 [Indexer]: now sync _k.fnm
> >> > IW 9 [Lucene Merge Thread #0]: now merge
> >> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
> >> > [mergeDocStores]
> >> > merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
> >> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> >> _k:C509->_j
> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
> >> _d:C1220->_d
> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> >> _j:C1456->_j
> >> > _k:C509->_j into _l [mergeDocStores]
> >> > IW 9 [Indexer]: now sync _k.tii
> >> > IW 9 [Indexer]: now sync _k.tis
> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
> >> > IW 9 [Indexer]: now sync _j.tis
> >> > IW 9 [Indexer]: now sync _j.tii
> >> > IW 9 [Indexer]: now sync _j.fdx
> >> > IW 9 [Indexer]: now sync _j.fdt
> >> > IW 9 [Indexer]: now sync _k.prx
> >> > IW 9 [Indexer]: now sync _k.nrm
> >> > IW 9 [Indexer]: now sync _j.nrm
> >> > IW 9 [Indexer]: now sync _j.prx
> >> > IW 9 [Indexer]: now sync _j.fnm
> >> > IW 9 [Indexer]: now sync _k.frq
> >> > IW 9 [Indexer]: now sync _j.frq
> >> > IW 9 [Indexer]: done all syncs
> >> > IW 9 [Indexer]: flush at getReader
> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k
> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> >> > numBufDelTerms=0
> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j
> >> > IW 9 [Indexer]: flush shared docStore segment _k
> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k
> >> numDocs=0
> >> > IW 9 [Indexer]: commit: start
> >> > IW 9 [Indexer]: commit: already prepared
> >> > IW 9 [Indexer]: commit: pendingCommit != null
> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit =
> >> true]
> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
> >> > IFD [Indexer]: delete "segments_b"
> >> > IW 9 [Indexer]: commit: done
> >> > IW 9 [Indexer]: flush at getReader
> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j
> >> > IW 9 [Indexer]: now flush at close
> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j
> >> > IW 9 [Indexer]: CMS: now merge
> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
> >> _d:C1220->_d
> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> >> _j:C1456->_j
> >> > _k:C509->_j
> >> > IW 9 [Indexer]: CMS: no more merges pending; now return
> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
> >> _g:C858->_f
> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
> >> _c:C717->_b
> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> >> _i:C703->_h
> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
> ;
> >> > isCommit = false]
> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
> ;
> >> > isCommit = false]
> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
> >> > IW 9 [Indexer]: now call final commit()
> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
> >> > IW 9 [Indexer]: now sync _l.nrm
> >> > IW 9 [Indexer]: now sync _l.fnm
> >> > IW 9 [Indexer]: now sync _l.prx
> >> > IW 9 [Indexer]: now sync _l.frq
> >> > IW 9 [Indexer]: now sync _l.tis
> >> > IW 9 [Indexer]: now sync _l.fdx
> >> > IW 9 [Indexer]: now sync _l.fdt
> >> > IW 9 [Indexer]: now sync _l.tii
> >> > IW 9 [Indexer]: done all syncs
> >> > IW 9 [Indexer]: commit: pendingCommit != null
> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit =
> true]
> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
> >> > IFD [Indexer]: delete "_h.prx"
> >> > IFD [Indexer]: delete "_f.tis"
> >> > IFD [Indexer]: delete "_g.tis"
> >> > IFD [Indexer]: delete "_d.tii"
> >> > IFD [Indexer]: delete "_e.frq"
> >> > IFD [Indexer]: delete "_k.fnm"
> >> > IFD [Indexer]: delete "_c.tii"
> >> > IFD [Indexer]: delete "_h.nrm"
> >> > IFD [Indexer]: delete "_i.tis"
> >> > IFD [Indexer]: delete "_k.tii"
> >> > IFD [Indexer]: delete "_h.fdt"
> >> > IFD [Indexer]: delete "_c.tis"
> >> > IFD [Indexer]: delete "_g.tii"
> >> > IFD [Indexer]: delete "_h.fdx"
> >> > IFD [Indexer]: delete "_c.nrm"
> >> > IFD [Indexer]: delete "_i.tii"
> >> > IFD [Indexer]: delete "_d.tis"
> >> > IFD [Indexer]: delete "_f.tii"
> >> > IFD [Indexer]: delete "_e.fnm"
> >> > IFD [Indexer]: delete "_k.tis"
> >> > IFD [Indexer]: delete "_d.frq"
> >> > IFD [Indexer]: delete "_f.nrm"
> >> > IFD [Indexer]: delete "_h.fnm"
> >> > IFD [Indexer]: delete "segments_c"
> >> > IFD [Indexer]: delete "_c.prx"
> >> > IFD [Indexer]: delete "_d.fnm"
> >> > IFD [Indexer]: delete "_h.frq"
> >> > IFD [Indexer]: delete "_h.tis"
> >> > IFD [Indexer]: delete "_f.fdt"
> >> > IFD [Indexer]: delete "_j.tis"
> >> > IFD [Indexer]: delete "_f.fdx"
> >> > IFD [Indexer]: delete "_g.nrm"
> >> > IFD [Indexer]: delete "_f.prx"
> >> > IFD [Indexer]: delete "_j.tii"
> >> > IFD [Indexer]: delete "_h.tii"
> >> > IFD [Indexer]: delete "_e.prx"
> >> > IFD [Indexer]: delete "_c.frq"
> >> > IFD [Indexer]: delete "_i.prx"
> >> > IFD [Indexer]: delete "_j.fdx"
> >> > IFD [Indexer]: delete "_i.nrm"
> >> > IFD [Indexer]: delete "_j.fdt"
> >> > IFD [Indexer]: delete "_k.prx"
> >> > IFD [Indexer]: delete "_b.nrm"
> >> > IFD [Indexer]: delete "_e.tis"
> >> > IFD [Indexer]: delete "_e.tii"
> >> > IFD [Indexer]: delete "_g.fnm"
> >> > IFD [Indexer]: delete "_k.nrm"
> >> > IFD [Indexer]: delete "_j.nrm"
> >> > IFD [Indexer]: delete "_j.prx"
> >> > IFD [Indexer]: delete "_e.nrm"
> >> > IFD [Indexer]: delete "_g.prx"
> >> > IFD [Indexer]: delete "_b.prx"
> >> > IFD [Indexer]: delete "_i.frq"
> >> > IFD [Indexer]: delete "_b.tis"
> >> > IFD [Indexer]: delete "_d.prx"
> >> > IFD [Indexer]: delete "_g.frq"
> >> > IFD [Indexer]: delete "_d.fdx"
> >> > IFD [Indexer]: delete "_b.tii"
> >> > IFD [Indexer]: delete "_b.frq"
> >> > IFD [Indexer]: delete "_d.fdt"
> >> > IFD [Indexer]: delete "_f.fnm"
> >> > IFD [Indexer]: delete "_j.fnm"
> >> > IFD [Indexer]: delete "_b.fnm"
> >> > IFD [Indexer]: delete "_d.nrm"
> >> > IFD [Indexer]: delete "_b.fdt"
> >> > IFD [Indexer]: delete "_k.frq"
> >> > IFD [Indexer]: delete "_c.fnm"
> >> > IFD [Indexer]: delete "_j.frq"
> >> > IFD [Indexer]: delete "_i.fnm"
> >> > IFD [Indexer]: delete "_f.frq"
> >> > IFD [Indexer]: delete "_b.fdx"
> >> > IW 9 [Indexer]: commit: done
> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
> >> >
> >> > =====================================================
> >> >
> >> > IFD [Indexer]: setInfoStream
> >> >
> >>
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
> >> > IW 10 [Indexer]: setInfoStream:
> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> >> > pathname>\lresumes1.search.main.1 autoCommit=false
> >> >
> >>
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
> >>
> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
> >> =16.0
> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909
> >> > IW 10 [Indexer]: flush at getReader
> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
> >> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16
> >> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
> flushDocStores=false
> >> > numDocs=1526 numBufDelTerms=1526
> >> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
> >> numDocs=1526
> >> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> >> > isCommit = false]
> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> >> > isCommit = false]
> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2
> >> segments
> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1
> >> segments
> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
> >> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909
> _m:C1526->_m
> >> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now return
> >> > IW 10 [Indexer]: prepareCommit: flush
> >> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m
> >> docStoreOffset=1526
> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
> >> > numBufDelTerms=501
> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> >> > IW 10 [Indexer]: flush shared docStore segment _m
> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
> >> > numDocs=2027
> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
> >> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
> >> > docs/MB=316.593 new/old=23.953%
> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
> >> false]
> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
> >> docIDs
> >> > and 0 deleted queries on 4 segments.
> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
> >> false]
> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments
> >> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
> >> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
> >> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
> >> > IW 10 [Indexer]: CMS: now merge
> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m
> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
> >> > _n:C501->_m changeCount=6
> >> > IW 10 [Indexer]: now sync _m.frq
> >> > IW 10 [Indexer]: now sync _m.prx
> >> > IW 10 [Indexer]: now sync _n.tis
> >> > IW 10 [Indexer]: now sync _m.tis
> >> > IW 10 [Indexer]: now sync _n.frq
> >> > IW 10 [Indexer]: now sync _n.tii
> >> > IW 10 [Indexer]: now sync _m.tii
> >> > IW 10 [Indexer]: now sync _m.fnm
> >> > IW 10 [Indexer]: now sync _n.fnm
> >> > IW 10 [Indexer]: now sync _n.nrm
> >> > IW 10 [Indexer]: now sync _n.prx
> >> > IW 10 [Indexer]: now sync _m.nrm
> >> > IW 10 [Indexer]: now sync _m.fdt
> >> > IW 10 [Indexer]: now sync _m.fdx
> >> > IW 10 [Indexer]: done all syncs
> >> > IW 10 [Indexer]: flush at getReader
> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> >> > numBufDelTerms=0
> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> >> > _n:C501->_m
> >> > IW 10 [Indexer]: flush shared docStore segment _n
> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n
> >> numDocs=0
> >> > IW 10 [Indexer]: commit: start
> >> > IW 10 [Indexer]: commit: already prepared
> >> > IW 10 [Indexer]: commit: pendingCommit != null
> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit =
> true]
> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
> >> > IFD [Indexer]: delete "segments_d"
> >> > IW 10 [Indexer]: commit: done
> >> > IW 10 [Indexer]: flush at getReader
> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> >> > _n:C501->_m
> >> > IW 10 [Indexer]: now flush at close
> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> >> > numBufDelTerms=0
> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> >> > _n:C501->_m
> >> > IW 10 [Indexer]: CMS: now merge
> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m
> >> > IW 10 [Indexer]: CMS: no more merges pending; now return
> >> > IW 10 [Indexer]: now call final commit()
> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> >> > IW 10 [Indexer]: skip startCommit(): no changes pending
> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip
> >> > IW 10 [Indexer]: commit: done
> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> >> >
> >> >
> >> > Peter
> >> >
> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
> >> > lucene@mikemccandless.com> wrote:
> >> >
> >> >> Do you know the place in the infoStream output where you got a reader
> >> >> with the wrong (unexplained extra +1) version? If so, can you post
> >> >> the infoStream output up to that point?
> >> >>
> >> >> Mike
> >> >>
> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <
> peterlkeegan@gmail.com>
> >> >> wrote:
> >> >> > I've reproduced this and I have a bunch of infoStream log files.
> Since
> >> >> the
> >> >> > messages have no timestamps, it's hard to tell where the relevant
> >> entries
> >> >> > are. What should I be looking for?
> >> >> >
> >> >> > Peter
> >> >> >
> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <
> peterlkeegan@gmail.com
> >> >> >wrote:
> >> >> >
> >> >> >> I'm pretty sure there are flushes and segment merges going on, but
> as
> >> >> you
> >> >> >> said, that shouldn't affect the version increment. I'll see what I
> >> can
> >> >> do to
> >> >> >> get infoStream output.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Peter
> >> >> >>
> >> >> >>
> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
> >> >> >> lucene@mikemccandless.com> wrote:
> >> >> >>
> >> >> >>> Well I'm at a loss then. The version should only increment on
> >> commit.
> >> >> >>>
> >> >> >>> Can you make it all happen when infoStream is on, and post back?
> >> >> >>>
> >> >> >>> Mike
> >> >> >>>
> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <
> >> peterlkeegan@gmail.com
> >> >> >
> >> >> >>> wrote:
> >> >> >>> > Only one writer thread and one writer process.
> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean
> create,
> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
> >> >> >>> >
> >> >> >>> > Peter
> >> >> >>> >
> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
> >> >> >>> > lucene@mikemccandless.com> wrote:
> >> >> >>> >
> >> >> >>> >> That's curious.
> >> >> >>> >>
> >> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first
> >> prepare,
> >> >> >>> >> since that will call prepareCommit internally) that this
> version
> >> >> >>> >> should increase.
> >> >> >>> >>
> >> >> >>> >> Is there only 1 thread doing this?
> >> >> >>> >>
> >> >> >>> >> Oh, and, are you passing false for autoCommit?
> >> >> >>> >>
> >> >> >>> >> Mike
> >> >> >>> >>
> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
> >> >> peterlkeegan@gmail.com
> >> >> >>> >
> >> >> >>> >> wrote:
> >> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which
> gets
> >> >> >>> repeated
> >> >> >>> >> at
> >> >> >>> >> > regular intervals:
> >> >> >>> >> >
> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer,
> false,
> >> >> >>> >> maxFieldLen)
> >> >> >>> >> > 4. writer.getReader().getVersion();
> >> >> >>> >> > 5. writer.prepareCommit();
> >> >> >>> >> > 6. writer.getReader().getVersion();
> >> >> >>> >> > 7. writer.commit();
> >> >> >>> >> > 8. writer.close();
> >> >> >>> >> >
> >> >> >>> >> > I'm using the version number to keep external data in synch
> >> with
> >> >> the
> >> >> >>> >> index.
> >> >> >>> >> > Usually, the version number from (6) is 1 greater than from
> (4)
> >> >> and
> >> >> >>> the
> >> >> >>> >> > version from (4) equals the version from the previous (6).
> At
> >> >> least
> >> >> >>> once
> >> >> >>> >> a
> >> >> >>> >> > day, however, the version from (4) is 1 greater than from
> the
> >> >> >>> previous
> >> >> >>> >> (6).
> >> >> >>> >> > What would explain this sporadic behavior of version
> numbers?
> >> >> >>> >> >
> >> >> >>> >> > Thanks,
> >> >> >>> >> > Peter
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>> >>
> >> >> ---------------------------------------------------------------------
> >> >> >>> >> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> >> >> >>> >> For additional commands, e-mail:
> >> java-user-help@lucene.apache.org
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >
> >> >> >>>
> >> >> >>>
> >> ---------------------------------------------------------------------
> >> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> >>> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>
> >> >>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
Note that it's a BG merge (not commit)...
You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API?
If you call that, then call .getReader().getVersion(), then close &
open the writer, I think (but you better test to be sure!) the next
.getReader().getVersion() should always match.
Mike
On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <pe...@gmail.com> wrote:
> Is there a way for the application to wait for the BG commit to finish
> before it calls IW.close? If so, would this prevent the extra version? The
> extra version causes the app. to think that the external data it committed
> is out of synch with the index, which requires the app to do extra
> processing to re-synch.
>
> Thanks,
> Peter
>
>
> On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> OK -- I can now see what happened.
>>
>> There was a merge still running, when you called IW.commit (Lucene
>> Merge Thread #0). Because IW.commit does not wait for BG merges to
>> finish, but IW.close does (by default), this means you'll pick up an
>> extra version whenever a merge is running when you call close.
>>
>> Mike
>>
>> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com>
>> wrote:
>> > I'm pretty sure this output occurred when the version number skipped +1.
>> > The line containing '============'. separates the close/open IndexWriter.
>> >
>> > IFD [Indexer]: setInfoStream
>> >
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
>> > IW 9 [Indexer]: setInfoStream:
>> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>> > pathname>\lresumes1.search.main.1 autoCommit=false
>> >
>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
>> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
>> =16.0
>> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
>> _d:C1220->_d
>> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> > IW 9 [Indexer]: flush at getReader
>> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
>> > allocMB=15.816 deletesMB=0.203 triggerMB=16
>> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
>> > docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
>> > numDocs=1456 numBufDelTerms=1456
>> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
>> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
>> _h:C1291->_h
>> > _i:C703->_h
>> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j numDocs=1456
>> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704
>> newFlushedSize=4969789
>> > docs/MB=307.202 new/old=29.966%
>> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>> > isCommit = false]
>> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
>> > isCommit = false]
>> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
>> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 segments
>> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 segments
>> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
>> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b
>> _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j
>> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
>> > IW 9 [Indexer]: prepareCommit: flush
>> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j
>> docStoreOffset=1456
>> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
>> > numBufDelTerms=509
>> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j
>> > IW 9 [Indexer]: flush shared docStore segment _j
>> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
>> > numDocs=1965
>> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
>> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
>> > docs/MB=287.727 new/old=24.788%
>> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>> false]
>> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
>> docIDs
>> > and 0 deleted queries on 11 segments.
>> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
>> false]
>> > IW 9 [Indexer]: LMP: findMerges: 11 segments
>> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
>> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
>> > IW 9 [Indexer]: LMP: 1 to 11: add this merge
>> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j [total 1 pending]
>> > IW 9 [Indexer]: CMS: now merge
>> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>> _d:C1220->_d
>> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> _j:C1456->_j
>> > _k:C509->_j
>> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b
>> _d:C1220->_d
>> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> _j:C1456->_j
>> > _k:C509->_j into _l [mergeDocStores]
>> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
>> > IW 9 [Indexer]: CMS: no more merges pending; now return
>> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j changeCount=7
>> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
>> > IW 9 [Indexer]: now sync _k.fnm
>> > IW 9 [Lucene Merge Thread #0]: now merge
>> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
>> > [mergeDocStores]
>> > merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
>> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
>> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
>> _k:C509->_j
>> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
>> _d:C1220->_d
>> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> _j:C1456->_j
>> > _k:C509->_j into _l [mergeDocStores]
>> > IW 9 [Indexer]: now sync _k.tii
>> > IW 9 [Indexer]: now sync _k.tis
>> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
>> > IW 9 [Indexer]: now sync _j.tis
>> > IW 9 [Indexer]: now sync _j.tii
>> > IW 9 [Indexer]: now sync _j.fdx
>> > IW 9 [Indexer]: now sync _j.fdt
>> > IW 9 [Indexer]: now sync _k.prx
>> > IW 9 [Indexer]: now sync _k.nrm
>> > IW 9 [Indexer]: now sync _j.nrm
>> > IW 9 [Indexer]: now sync _j.prx
>> > IW 9 [Indexer]: now sync _j.fnm
>> > IW 9 [Indexer]: now sync _k.frq
>> > IW 9 [Indexer]: now sync _j.frq
>> > IW 9 [Indexer]: done all syncs
>> > IW 9 [Indexer]: flush at getReader
>> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>> > numBufDelTerms=0
>> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j
>> > IW 9 [Indexer]: flush shared docStore segment _k
>> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k
>> numDocs=0
>> > IW 9 [Indexer]: commit: start
>> > IW 9 [Indexer]: commit: already prepared
>> > IW 9 [Indexer]: commit: pendingCommit != null
>> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
>> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit =
>> true]
>> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
>> > IFD [Indexer]: delete "segments_b"
>> > IW 9 [Indexer]: commit: done
>> > IW 9 [Indexer]: flush at getReader
>> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j
>> > IW 9 [Indexer]: now flush at close
>> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j
>> > IW 9 [Indexer]: CMS: now merge
>> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
>> _d:C1220->_d
>> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>> _j:C1456->_j
>> > _k:C509->_j
>> > IW 9 [Indexer]: CMS: no more merges pending; now return
>> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
>> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>> _g:C858->_f
>> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
>> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
>> _c:C717->_b
>> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>> _i:C703->_h
>> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
>> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
>> > isCommit = false]
>> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
>> > isCommit = false]
>> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
>> > IW 9 [Indexer]: now call final commit()
>> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
>> > IW 9 [Indexer]: now sync _l.nrm
>> > IW 9 [Indexer]: now sync _l.fnm
>> > IW 9 [Indexer]: now sync _l.prx
>> > IW 9 [Indexer]: now sync _l.frq
>> > IW 9 [Indexer]: now sync _l.tis
>> > IW 9 [Indexer]: now sync _l.fdx
>> > IW 9 [Indexer]: now sync _l.fdt
>> > IW 9 [Indexer]: now sync _l.tii
>> > IW 9 [Indexer]: done all syncs
>> > IW 9 [Indexer]: commit: pendingCommit != null
>> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
>> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = true]
>> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
>> > IFD [Indexer]: delete "_h.prx"
>> > IFD [Indexer]: delete "_f.tis"
>> > IFD [Indexer]: delete "_g.tis"
>> > IFD [Indexer]: delete "_d.tii"
>> > IFD [Indexer]: delete "_e.frq"
>> > IFD [Indexer]: delete "_k.fnm"
>> > IFD [Indexer]: delete "_c.tii"
>> > IFD [Indexer]: delete "_h.nrm"
>> > IFD [Indexer]: delete "_i.tis"
>> > IFD [Indexer]: delete "_k.tii"
>> > IFD [Indexer]: delete "_h.fdt"
>> > IFD [Indexer]: delete "_c.tis"
>> > IFD [Indexer]: delete "_g.tii"
>> > IFD [Indexer]: delete "_h.fdx"
>> > IFD [Indexer]: delete "_c.nrm"
>> > IFD [Indexer]: delete "_i.tii"
>> > IFD [Indexer]: delete "_d.tis"
>> > IFD [Indexer]: delete "_f.tii"
>> > IFD [Indexer]: delete "_e.fnm"
>> > IFD [Indexer]: delete "_k.tis"
>> > IFD [Indexer]: delete "_d.frq"
>> > IFD [Indexer]: delete "_f.nrm"
>> > IFD [Indexer]: delete "_h.fnm"
>> > IFD [Indexer]: delete "segments_c"
>> > IFD [Indexer]: delete "_c.prx"
>> > IFD [Indexer]: delete "_d.fnm"
>> > IFD [Indexer]: delete "_h.frq"
>> > IFD [Indexer]: delete "_h.tis"
>> > IFD [Indexer]: delete "_f.fdt"
>> > IFD [Indexer]: delete "_j.tis"
>> > IFD [Indexer]: delete "_f.fdx"
>> > IFD [Indexer]: delete "_g.nrm"
>> > IFD [Indexer]: delete "_f.prx"
>> > IFD [Indexer]: delete "_j.tii"
>> > IFD [Indexer]: delete "_h.tii"
>> > IFD [Indexer]: delete "_e.prx"
>> > IFD [Indexer]: delete "_c.frq"
>> > IFD [Indexer]: delete "_i.prx"
>> > IFD [Indexer]: delete "_j.fdx"
>> > IFD [Indexer]: delete "_i.nrm"
>> > IFD [Indexer]: delete "_j.fdt"
>> > IFD [Indexer]: delete "_k.prx"
>> > IFD [Indexer]: delete "_b.nrm"
>> > IFD [Indexer]: delete "_e.tis"
>> > IFD [Indexer]: delete "_e.tii"
>> > IFD [Indexer]: delete "_g.fnm"
>> > IFD [Indexer]: delete "_k.nrm"
>> > IFD [Indexer]: delete "_j.nrm"
>> > IFD [Indexer]: delete "_j.prx"
>> > IFD [Indexer]: delete "_e.nrm"
>> > IFD [Indexer]: delete "_g.prx"
>> > IFD [Indexer]: delete "_b.prx"
>> > IFD [Indexer]: delete "_i.frq"
>> > IFD [Indexer]: delete "_b.tis"
>> > IFD [Indexer]: delete "_d.prx"
>> > IFD [Indexer]: delete "_g.frq"
>> > IFD [Indexer]: delete "_d.fdx"
>> > IFD [Indexer]: delete "_b.tii"
>> > IFD [Indexer]: delete "_b.frq"
>> > IFD [Indexer]: delete "_d.fdt"
>> > IFD [Indexer]: delete "_f.fnm"
>> > IFD [Indexer]: delete "_j.fnm"
>> > IFD [Indexer]: delete "_b.fnm"
>> > IFD [Indexer]: delete "_d.nrm"
>> > IFD [Indexer]: delete "_b.fdt"
>> > IFD [Indexer]: delete "_k.frq"
>> > IFD [Indexer]: delete "_c.fnm"
>> > IFD [Indexer]: delete "_j.frq"
>> > IFD [Indexer]: delete "_i.fnm"
>> > IFD [Indexer]: delete "_f.frq"
>> > IFD [Indexer]: delete "_b.fdx"
>> > IW 9 [Indexer]: commit: done
>> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
>> >
>> > =====================================================
>> >
>> > IFD [Indexer]: setInfoStream
>> >
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
>> > IW 10 [Indexer]: setInfoStream:
>> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>> > pathname>\lresumes1.search.main.1 autoCommit=false
>> >
>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
>> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
>> =16.0
>> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>> > maxFieldLength=100000 index=_a:C9780 _l:C9909
>> > IW 10 [Indexer]: flush at getReader
>> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
>> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
>> > allocMB=15.789 deletesMB=0.212 triggerMB=16
>> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
>> > docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
>> > numDocs=1526 numBufDelTerms=1526
>> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
>> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
>> numDocs=1526
>> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
>> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
>> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>> > isCommit = false]
>> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
>> > isCommit = false]
>> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
>> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2
>> segments
>> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1
>> segments
>> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
>> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
>> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now return
>> > IW 10 [Indexer]: prepareCommit: flush
>> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m
>> docStoreOffset=1526
>> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
>> > numBufDelTerms=501
>> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> > IW 10 [Indexer]: flush shared docStore segment _m
>> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
>> > numDocs=2027
>> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
>> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
>> > docs/MB=316.593 new/old=23.953%
>> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>> false]
>> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
>> docIDs
>> > and 0 deleted queries on 4 segments.
>> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
>> false]
>> > IW 10 [Indexer]: LMP: findMerges: 4 segments
>> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
>> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
>> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
>> > IW 10 [Indexer]: CMS: now merge
>> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>> > IW 10 [Indexer]: CMS: no more merges pending; now return
>> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
>> > _n:C501->_m changeCount=6
>> > IW 10 [Indexer]: now sync _m.frq
>> > IW 10 [Indexer]: now sync _m.prx
>> > IW 10 [Indexer]: now sync _n.tis
>> > IW 10 [Indexer]: now sync _m.tis
>> > IW 10 [Indexer]: now sync _n.frq
>> > IW 10 [Indexer]: now sync _n.tii
>> > IW 10 [Indexer]: now sync _m.tii
>> > IW 10 [Indexer]: now sync _m.fnm
>> > IW 10 [Indexer]: now sync _n.fnm
>> > IW 10 [Indexer]: now sync _n.nrm
>> > IW 10 [Indexer]: now sync _n.prx
>> > IW 10 [Indexer]: now sync _m.nrm
>> > IW 10 [Indexer]: now sync _m.fdt
>> > IW 10 [Indexer]: now sync _m.fdx
>> > IW 10 [Indexer]: done all syncs
>> > IW 10 [Indexer]: flush at getReader
>> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>> > numBufDelTerms=0
>> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> > _n:C501->_m
>> > IW 10 [Indexer]: flush shared docStore segment _n
>> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n
>> numDocs=0
>> > IW 10 [Indexer]: commit: start
>> > IW 10 [Indexer]: commit: already prepared
>> > IW 10 [Indexer]: commit: pendingCommit != null
>> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
>> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = true]
>> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
>> > IFD [Indexer]: delete "segments_d"
>> > IW 10 [Indexer]: commit: done
>> > IW 10 [Indexer]: flush at getReader
>> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> > _n:C501->_m
>> > IW 10 [Indexer]: now flush at close
>> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>> > numBufDelTerms=0
>> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
>> > _n:C501->_m
>> > IW 10 [Indexer]: CMS: now merge
>> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>> > IW 10 [Indexer]: CMS: no more merges pending; now return
>> > IW 10 [Indexer]: now call final commit()
>> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>> > IW 10 [Indexer]: skip startCommit(): no changes pending
>> > IW 10 [Indexer]: commit: pendingCommit == null; skip
>> > IW 10 [Indexer]: commit: done
>> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>> >
>> >
>> > Peter
>> >
>> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
>> > lucene@mikemccandless.com> wrote:
>> >
>> >> Do you know the place in the infoStream output where you got a reader
>> >> with the wrong (unexplained extra +1) version? If so, can you post
>> >> the infoStream output up to that point?
>> >>
>> >> Mike
>> >>
>> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <pe...@gmail.com>
>> >> wrote:
>> >> > I've reproduced this and I have a bunch of infoStream log files. Since
>> >> the
>> >> > messages have no timestamps, it's hard to tell where the relevant
>> entries
>> >> > are. What should I be looking for?
>> >> >
>> >> > Peter
>> >> >
>> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <peterlkeegan@gmail.com
>> >> >wrote:
>> >> >
>> >> >> I'm pretty sure there are flushes and segment merges going on, but as
>> >> you
>> >> >> said, that shouldn't affect the version increment. I'll see what I
>> can
>> >> do to
>> >> >> get infoStream output.
>> >> >>
>> >> >> Thanks,
>> >> >> Peter
>> >> >>
>> >> >>
>> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
>> >> >> lucene@mikemccandless.com> wrote:
>> >> >>
>> >> >>> Well I'm at a loss then. The version should only increment on
>> commit.
>> >> >>>
>> >> >>> Can you make it all happen when infoStream is on, and post back?
>> >> >>>
>> >> >>> Mike
>> >> >>>
>> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <
>> peterlkeegan@gmail.com
>> >> >
>> >> >>> wrote:
>> >> >>> > Only one writer thread and one writer process.
>> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
>> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
>> >> >>> >
>> >> >>> > Peter
>> >> >>> >
>> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>> >> >>> > lucene@mikemccandless.com> wrote:
>> >> >>> >
>> >> >>> >> That's curious.
>> >> >>> >>
>> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first
>> prepare,
>> >> >>> >> since that will call prepareCommit internally) that this version
>> >> >>> >> should increase.
>> >> >>> >>
>> >> >>> >> Is there only 1 thread doing this?
>> >> >>> >>
>> >> >>> >> Oh, and, are you passing false for autoCommit?
>> >> >>> >>
>> >> >>> >> Mike
>> >> >>> >>
>> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
>> >> peterlkeegan@gmail.com
>> >> >>> >
>> >> >>> >> wrote:
>> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
>> >> >>> repeated
>> >> >>> >> at
>> >> >>> >> > regular intervals:
>> >> >>> >> >
>> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
>> >> >>> >> maxFieldLen)
>> >> >>> >> > 4. writer.getReader().getVersion();
>> >> >>> >> > 5. writer.prepareCommit();
>> >> >>> >> > 6. writer.getReader().getVersion();
>> >> >>> >> > 7. writer.commit();
>> >> >>> >> > 8. writer.close();
>> >> >>> >> >
>> >> >>> >> > I'm using the version number to keep external data in synch
>> with
>> >> the
>> >> >>> >> index.
>> >> >>> >> > Usually, the version number from (6) is 1 greater than from (4)
>> >> and
>> >> >>> the
>> >> >>> >> > version from (4) equals the version from the previous (6). At
>> >> least
>> >> >>> once
>> >> >>> >> a
>> >> >>> >> > day, however, the version from (4) is 1 greater than from the
>> >> >>> previous
>> >> >>> >> (6).
>> >> >>> >> > What would explain this sporadic behavior of version numbers?
>> >> >>> >> >
>> >> >>> >> > Thanks,
>> >> >>> >> > Peter
>> >> >>> >> >
>> >> >>> >>
>> >> >>> >>
>> >> ---------------------------------------------------------------------
>> >> >>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> >>> >> For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>> >> >>> >>
>> >> >>> >>
>> >> >>> >
>> >> >>>
>> >> >>>
>> ---------------------------------------------------------------------
>> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
Is there a way for the application to wait for the BG commit to finish
before it calls IW.close? If so, would this prevent the extra version? The
extra version causes the app. to think that the external data it committed
is out of synch with the index, which requires the app to do extra
processing to re-synch.
Thanks,
Peter
On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> OK -- I can now see what happened.
>
> There was a merge still running, when you called IW.commit (Lucene
> Merge Thread #0). Because IW.commit does not wait for BG merges to
> finish, but IW.close does (by default), this means you'll pick up an
> extra version whenever a merge is running when you call close.
>
> Mike
>
> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com>
> wrote:
> > I'm pretty sure this output occurred when the version number skipped +1.
> > The line containing '============'. separates the close/open IndexWriter.
> >
> > IFD [Indexer]: setInfoStream
> >
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
> > IW 9 [Indexer]: setInfoStream:
> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> > pathname>\lresumes1.search.main.1 autoCommit=false
> >
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
> =16.0
> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d
> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> > IW 9 [Indexer]: flush at getReader
> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
> > allocMB=15.816 deletesMB=0.203 triggerMB=16
> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
> > docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
> > numDocs=1456 numBufDelTerms=1456
> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
> _h:C1291->_h
> > _i:C703->_h
> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j numDocs=1456
> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704
> newFlushedSize=4969789
> > docs/MB=307.202 new/old=29.966%
> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> > isCommit = false]
> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> > isCommit = false]
> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 segments
> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 segments
> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b
> _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j
> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
> > IW 9 [Indexer]: prepareCommit: flush
> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j
> docStoreOffset=1456
> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
> > numBufDelTerms=509
> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j
> > IW 9 [Indexer]: flush shared docStore segment _j
> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
> > numDocs=1965
> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
> > docs/MB=287.727 new/old=24.788%
> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
> false]
> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
> docIDs
> > and 0 deleted queries on 11 segments.
> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit =
> false]
> > IW 9 [Indexer]: LMP: findMerges: 11 segments
> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
> > IW 9 [Indexer]: LMP: 1 to 11: add this merge
> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j [total 1 pending]
> > IW 9 [Indexer]: CMS: now merge
> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d
> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> > _k:C509->_j
> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b
> _d:C1220->_d
> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> > _k:C509->_j into _l [mergeDocStores]
> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
> > IW 9 [Indexer]: CMS: no more merges pending; now return
> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j changeCount=7
> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
> > IW 9 [Indexer]: now sync _k.fnm
> > IW 9 [Lucene Merge Thread #0]: now merge
> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
> > [mergeDocStores]
> > merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> _k:C509->_j
> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
> _d:C1220->_d
> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> > _k:C509->_j into _l [mergeDocStores]
> > IW 9 [Indexer]: now sync _k.tii
> > IW 9 [Indexer]: now sync _k.tis
> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
> > IW 9 [Indexer]: now sync _j.tis
> > IW 9 [Indexer]: now sync _j.tii
> > IW 9 [Indexer]: now sync _j.fdx
> > IW 9 [Indexer]: now sync _j.fdt
> > IW 9 [Indexer]: now sync _k.prx
> > IW 9 [Indexer]: now sync _k.nrm
> > IW 9 [Indexer]: now sync _j.nrm
> > IW 9 [Indexer]: now sync _j.prx
> > IW 9 [Indexer]: now sync _j.fnm
> > IW 9 [Indexer]: now sync _k.frq
> > IW 9 [Indexer]: now sync _j.frq
> > IW 9 [Indexer]: done all syncs
> > IW 9 [Indexer]: flush at getReader
> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> > numBufDelTerms=0
> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j
> > IW 9 [Indexer]: flush shared docStore segment _k
> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k
> numDocs=0
> > IW 9 [Indexer]: commit: start
> > IW 9 [Indexer]: commit: already prepared
> > IW 9 [Indexer]: commit: pendingCommit != null
> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit =
> true]
> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
> > IFD [Indexer]: delete "segments_b"
> > IW 9 [Indexer]: commit: done
> > IW 9 [Indexer]: flush at getReader
> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j
> > IW 9 [Indexer]: now flush at close
> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j
> > IW 9 [Indexer]: CMS: now merge
> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d
> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> > _k:C509->_j
> > IW 9 [Indexer]: CMS: no more merges pending; now return
> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
> _g:C858->_f
> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
> _c:C717->_b
> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
> > isCommit = false]
> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
> > isCommit = false]
> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
> > IW 9 [Indexer]: now call final commit()
> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
> > IW 9 [Indexer]: now sync _l.nrm
> > IW 9 [Indexer]: now sync _l.fnm
> > IW 9 [Indexer]: now sync _l.prx
> > IW 9 [Indexer]: now sync _l.frq
> > IW 9 [Indexer]: now sync _l.tis
> > IW 9 [Indexer]: now sync _l.fdx
> > IW 9 [Indexer]: now sync _l.fdt
> > IW 9 [Indexer]: now sync _l.tii
> > IW 9 [Indexer]: done all syncs
> > IW 9 [Indexer]: commit: pendingCommit != null
> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = true]
> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
> > IFD [Indexer]: delete "_h.prx"
> > IFD [Indexer]: delete "_f.tis"
> > IFD [Indexer]: delete "_g.tis"
> > IFD [Indexer]: delete "_d.tii"
> > IFD [Indexer]: delete "_e.frq"
> > IFD [Indexer]: delete "_k.fnm"
> > IFD [Indexer]: delete "_c.tii"
> > IFD [Indexer]: delete "_h.nrm"
> > IFD [Indexer]: delete "_i.tis"
> > IFD [Indexer]: delete "_k.tii"
> > IFD [Indexer]: delete "_h.fdt"
> > IFD [Indexer]: delete "_c.tis"
> > IFD [Indexer]: delete "_g.tii"
> > IFD [Indexer]: delete "_h.fdx"
> > IFD [Indexer]: delete "_c.nrm"
> > IFD [Indexer]: delete "_i.tii"
> > IFD [Indexer]: delete "_d.tis"
> > IFD [Indexer]: delete "_f.tii"
> > IFD [Indexer]: delete "_e.fnm"
> > IFD [Indexer]: delete "_k.tis"
> > IFD [Indexer]: delete "_d.frq"
> > IFD [Indexer]: delete "_f.nrm"
> > IFD [Indexer]: delete "_h.fnm"
> > IFD [Indexer]: delete "segments_c"
> > IFD [Indexer]: delete "_c.prx"
> > IFD [Indexer]: delete "_d.fnm"
> > IFD [Indexer]: delete "_h.frq"
> > IFD [Indexer]: delete "_h.tis"
> > IFD [Indexer]: delete "_f.fdt"
> > IFD [Indexer]: delete "_j.tis"
> > IFD [Indexer]: delete "_f.fdx"
> > IFD [Indexer]: delete "_g.nrm"
> > IFD [Indexer]: delete "_f.prx"
> > IFD [Indexer]: delete "_j.tii"
> > IFD [Indexer]: delete "_h.tii"
> > IFD [Indexer]: delete "_e.prx"
> > IFD [Indexer]: delete "_c.frq"
> > IFD [Indexer]: delete "_i.prx"
> > IFD [Indexer]: delete "_j.fdx"
> > IFD [Indexer]: delete "_i.nrm"
> > IFD [Indexer]: delete "_j.fdt"
> > IFD [Indexer]: delete "_k.prx"
> > IFD [Indexer]: delete "_b.nrm"
> > IFD [Indexer]: delete "_e.tis"
> > IFD [Indexer]: delete "_e.tii"
> > IFD [Indexer]: delete "_g.fnm"
> > IFD [Indexer]: delete "_k.nrm"
> > IFD [Indexer]: delete "_j.nrm"
> > IFD [Indexer]: delete "_j.prx"
> > IFD [Indexer]: delete "_e.nrm"
> > IFD [Indexer]: delete "_g.prx"
> > IFD [Indexer]: delete "_b.prx"
> > IFD [Indexer]: delete "_i.frq"
> > IFD [Indexer]: delete "_b.tis"
> > IFD [Indexer]: delete "_d.prx"
> > IFD [Indexer]: delete "_g.frq"
> > IFD [Indexer]: delete "_d.fdx"
> > IFD [Indexer]: delete "_b.tii"
> > IFD [Indexer]: delete "_b.frq"
> > IFD [Indexer]: delete "_d.fdt"
> > IFD [Indexer]: delete "_f.fnm"
> > IFD [Indexer]: delete "_j.fnm"
> > IFD [Indexer]: delete "_b.fnm"
> > IFD [Indexer]: delete "_d.nrm"
> > IFD [Indexer]: delete "_b.fdt"
> > IFD [Indexer]: delete "_k.frq"
> > IFD [Indexer]: delete "_c.fnm"
> > IFD [Indexer]: delete "_j.frq"
> > IFD [Indexer]: delete "_i.fnm"
> > IFD [Indexer]: delete "_f.frq"
> > IFD [Indexer]: delete "_b.fdx"
> > IW 9 [Indexer]: commit: done
> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
> >
> > =====================================================
> >
> > IFD [Indexer]: setInfoStream
> >
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
> > IW 10 [Indexer]: setInfoStream:
> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> > pathname>\lresumes1.search.main.1 autoCommit=false
> >
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
> =16.0
> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> > maxFieldLength=100000 index=_a:C9780 _l:C9909
> > IW 10 [Indexer]: flush at getReader
> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
> > allocMB=15.789 deletesMB=0.212 triggerMB=16
> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
> > docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
> > numDocs=1526 numBufDelTerms=1526
> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
> numDocs=1526
> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> > isCommit = false]
> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> > isCommit = false]
> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2
> segments
> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1
> segments
> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now return
> > IW 10 [Indexer]: prepareCommit: flush
> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m
> docStoreOffset=1526
> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
> > numBufDelTerms=501
> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> > IW 10 [Indexer]: flush shared docStore segment _m
> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
> > numDocs=2027
> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
> > docs/MB=316.593 new/old=23.953%
> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
> false]
> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
> docIDs
> > and 0 deleted queries on 4 segments.
> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit =
> false]
> > IW 10 [Indexer]: LMP: findMerges: 4 segments
> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
> > IW 10 [Indexer]: CMS: now merge
> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> > IW 10 [Indexer]: CMS: no more merges pending; now return
> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
> > _n:C501->_m changeCount=6
> > IW 10 [Indexer]: now sync _m.frq
> > IW 10 [Indexer]: now sync _m.prx
> > IW 10 [Indexer]: now sync _n.tis
> > IW 10 [Indexer]: now sync _m.tis
> > IW 10 [Indexer]: now sync _n.frq
> > IW 10 [Indexer]: now sync _n.tii
> > IW 10 [Indexer]: now sync _m.tii
> > IW 10 [Indexer]: now sync _m.fnm
> > IW 10 [Indexer]: now sync _n.fnm
> > IW 10 [Indexer]: now sync _n.nrm
> > IW 10 [Indexer]: now sync _n.prx
> > IW 10 [Indexer]: now sync _m.nrm
> > IW 10 [Indexer]: now sync _m.fdt
> > IW 10 [Indexer]: now sync _m.fdx
> > IW 10 [Indexer]: done all syncs
> > IW 10 [Indexer]: flush at getReader
> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> > numBufDelTerms=0
> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> > _n:C501->_m
> > IW 10 [Indexer]: flush shared docStore segment _n
> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n
> numDocs=0
> > IW 10 [Indexer]: commit: start
> > IW 10 [Indexer]: commit: already prepared
> > IW 10 [Indexer]: commit: pendingCommit != null
> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = true]
> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
> > IFD [Indexer]: delete "segments_d"
> > IW 10 [Indexer]: commit: done
> > IW 10 [Indexer]: flush at getReader
> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> > _n:C501->_m
> > IW 10 [Indexer]: now flush at close
> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null
> docStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> > _n:C501->_m
> > IW 10 [Indexer]: CMS: now merge
> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> > IW 10 [Indexer]: CMS: no more merges pending; now return
> > IW 10 [Indexer]: now call final commit()
> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> > IW 10 [Indexer]: skip startCommit(): no changes pending
> > IW 10 [Indexer]: commit: pendingCommit == null; skip
> > IW 10 [Indexer]: commit: done
> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> >
> >
> > Peter
> >
> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> Do you know the place in the infoStream output where you got a reader
> >> with the wrong (unexplained extra +1) version? If so, can you post
> >> the infoStream output up to that point?
> >>
> >> Mike
> >>
> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <pe...@gmail.com>
> >> wrote:
> >> > I've reproduced this and I have a bunch of infoStream log files. Since
> >> the
> >> > messages have no timestamps, it's hard to tell where the relevant
> entries
> >> > are. What should I be looking for?
> >> >
> >> > Peter
> >> >
> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <peterlkeegan@gmail.com
> >> >wrote:
> >> >
> >> >> I'm pretty sure there are flushes and segment merges going on, but as
> >> you
> >> >> said, that shouldn't affect the version increment. I'll see what I
> can
> >> do to
> >> >> get infoStream output.
> >> >>
> >> >> Thanks,
> >> >> Peter
> >> >>
> >> >>
> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
> >> >> lucene@mikemccandless.com> wrote:
> >> >>
> >> >>> Well I'm at a loss then. The version should only increment on
> commit.
> >> >>>
> >> >>> Can you make it all happen when infoStream is on, and post back?
> >> >>>
> >> >>> Mike
> >> >>>
> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <
> peterlkeegan@gmail.com
> >> >
> >> >>> wrote:
> >> >>> > Only one writer thread and one writer process.
> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
> >> >>> >
> >> >>> > Peter
> >> >>> >
> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
> >> >>> > lucene@mikemccandless.com> wrote:
> >> >>> >
> >> >>> >> That's curious.
> >> >>> >>
> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first
> prepare,
> >> >>> >> since that will call prepareCommit internally) that this version
> >> >>> >> should increase.
> >> >>> >>
> >> >>> >> Is there only 1 thread doing this?
> >> >>> >>
> >> >>> >> Oh, and, are you passing false for autoCommit?
> >> >>> >>
> >> >>> >> Mike
> >> >>> >>
> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
> >> peterlkeegan@gmail.com
> >> >>> >
> >> >>> >> wrote:
> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
> >> >>> repeated
> >> >>> >> at
> >> >>> >> > regular intervals:
> >> >>> >> >
> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
> >> >>> >> maxFieldLen)
> >> >>> >> > 4. writer.getReader().getVersion();
> >> >>> >> > 5. writer.prepareCommit();
> >> >>> >> > 6. writer.getReader().getVersion();
> >> >>> >> > 7. writer.commit();
> >> >>> >> > 8. writer.close();
> >> >>> >> >
> >> >>> >> > I'm using the version number to keep external data in synch
> with
> >> the
> >> >>> >> index.
> >> >>> >> > Usually, the version number from (6) is 1 greater than from (4)
> >> and
> >> >>> the
> >> >>> >> > version from (4) equals the version from the previous (6). At
> >> least
> >> >>> once
> >> >>> >> a
> >> >>> >> > day, however, the version from (4) is 1 greater than from the
> >> >>> previous
> >> >>> >> (6).
> >> >>> >> > What would explain this sporadic behavior of version numbers?
> >> >>> >> >
> >> >>> >> > Thanks,
> >> >>> >> > Peter
> >> >>> >> >
> >> >>> >>
> >> >>> >>
> >> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >>> >> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK -- I can now see what happened.
There was a merge still running, when you called IW.commit (Lucene
Merge Thread #0). Because IW.commit does not wait for BG merges to
finish, but IW.close does (by default), this means you'll pick up an
extra version whenever a merge is running when you call close.
Mike
On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <pe...@gmail.com> wrote:
> I'm pretty sure this output occurred when the version number skipped +1.
> The line containing '============'. separates the close/open IndexWriter.
>
> IFD [Indexer]: setInfoStream
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
> IW 9 [Indexer]: setInfoStream:
> dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> pathname>\lresumes1.search.main.1 autoCommit=false
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB=16.0
> maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
> _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> IW 9 [Indexer]: flush at getReader
> IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
> allocMB=15.816 deletesMB=0.203 triggerMB=16
> IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
> docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
> numDocs=1456 numBufDelTerms=1456
> IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
> _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
> _i:C703->_h
> IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j numDocs=1456
> IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704 newFlushedSize=4969789
> docs/MB=307.202 new/old=29.966%
> IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> isCommit = false]
> IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
> isCommit = false]
> IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
> IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 segments
> IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 segments
> IW 9 [UpdWriterBuild : 9]: CMS: now merge
> IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
> IW 9 [Indexer]: prepareCommit: flush
> IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j docStoreOffset=1456
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
> numBufDelTerms=509
> IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j
> IW 9 [Indexer]: flush shared docStore segment _j
> IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
> numDocs=1965
> IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
> IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
> docs/MB=287.727 new/old=24.788%
> IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = false]
> IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted docIDs
> and 0 deleted queries on 11 segments.
> IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = false]
> IW 9 [Indexer]: LMP: findMerges: 11 segments
> IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
> IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
> IW 9 [Indexer]: LMP: 1 to 11: add this merge
> IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j [total 1 pending]
> IW 9 [Indexer]: CMS: now merge
> IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
> _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> _k:C509->_j
> IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b _d:C1220->_d
> _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> _k:C509->_j into _l [mergeDocStores]
> IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
> IW 9 [Indexer]: CMS: no more merges pending; now return
> IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j changeCount=7
> IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
> IW 9 [Indexer]: now sync _k.fnm
> IW 9 [Lucene Merge Thread #0]: now merge
> merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
> _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
> [mergeDocStores]
> merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
> index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
> _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
> IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b _d:C1220->_d
> _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> _k:C509->_j into _l [mergeDocStores]
> IW 9 [Indexer]: now sync _k.tii
> IW 9 [Indexer]: now sync _k.tis
> IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
> IW 9 [Indexer]: now sync _j.tis
> IW 9 [Indexer]: now sync _j.tii
> IW 9 [Indexer]: now sync _j.fdx
> IW 9 [Indexer]: now sync _j.fdt
> IW 9 [Indexer]: now sync _k.prx
> IW 9 [Indexer]: now sync _k.nrm
> IW 9 [Indexer]: now sync _j.nrm
> IW 9 [Indexer]: now sync _j.prx
> IW 9 [Indexer]: now sync _j.fnm
> IW 9 [Indexer]: now sync _k.frq
> IW 9 [Indexer]: now sync _j.frq
> IW 9 [Indexer]: done all syncs
> IW 9 [Indexer]: flush at getReader
> IW 9 [Indexer]: flush: segment=null docStoreSegment=_k docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> numBufDelTerms=0
> IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j
> IW 9 [Indexer]: flush shared docStore segment _k
> IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k numDocs=0
> IW 9 [Indexer]: commit: start
> IW 9 [Indexer]: commit: already prepared
> IW 9 [Indexer]: commit: pendingCommit != null
> IW 9 [Indexer]: commit: wrote segments file "segments_c"
> IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit = true]
> IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
> IFD [Indexer]: delete "segments_b"
> IW 9 [Indexer]: commit: done
> IW 9 [Indexer]: flush at getReader
> IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j
> IW 9 [Indexer]: now flush at close
> IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j
> IW 9 [Indexer]: CMS: now merge
> IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
> _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
> _k:C509->_j
> IW 9 [Indexer]: CMS: no more merges pending; now return
> IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
> _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
> _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
> IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b _c:C717->_b
> _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
> _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
> IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
> isCommit = false]
> IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
> isCommit = false]
> IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
> IW 9 [Indexer]: now call final commit()
> IW 9 [Indexer]: startCommit(): start sizeInBytes=0
> IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
> IW 9 [Indexer]: now sync _l.nrm
> IW 9 [Indexer]: now sync _l.fnm
> IW 9 [Indexer]: now sync _l.prx
> IW 9 [Indexer]: now sync _l.frq
> IW 9 [Indexer]: now sync _l.tis
> IW 9 [Indexer]: now sync _l.fdx
> IW 9 [Indexer]: now sync _l.fdt
> IW 9 [Indexer]: now sync _l.tii
> IW 9 [Indexer]: done all syncs
> IW 9 [Indexer]: commit: pendingCommit != null
> IW 9 [Indexer]: commit: wrote segments file "segments_d"
> IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = true]
> IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
> IFD [Indexer]: delete "_h.prx"
> IFD [Indexer]: delete "_f.tis"
> IFD [Indexer]: delete "_g.tis"
> IFD [Indexer]: delete "_d.tii"
> IFD [Indexer]: delete "_e.frq"
> IFD [Indexer]: delete "_k.fnm"
> IFD [Indexer]: delete "_c.tii"
> IFD [Indexer]: delete "_h.nrm"
> IFD [Indexer]: delete "_i.tis"
> IFD [Indexer]: delete "_k.tii"
> IFD [Indexer]: delete "_h.fdt"
> IFD [Indexer]: delete "_c.tis"
> IFD [Indexer]: delete "_g.tii"
> IFD [Indexer]: delete "_h.fdx"
> IFD [Indexer]: delete "_c.nrm"
> IFD [Indexer]: delete "_i.tii"
> IFD [Indexer]: delete "_d.tis"
> IFD [Indexer]: delete "_f.tii"
> IFD [Indexer]: delete "_e.fnm"
> IFD [Indexer]: delete "_k.tis"
> IFD [Indexer]: delete "_d.frq"
> IFD [Indexer]: delete "_f.nrm"
> IFD [Indexer]: delete "_h.fnm"
> IFD [Indexer]: delete "segments_c"
> IFD [Indexer]: delete "_c.prx"
> IFD [Indexer]: delete "_d.fnm"
> IFD [Indexer]: delete "_h.frq"
> IFD [Indexer]: delete "_h.tis"
> IFD [Indexer]: delete "_f.fdt"
> IFD [Indexer]: delete "_j.tis"
> IFD [Indexer]: delete "_f.fdx"
> IFD [Indexer]: delete "_g.nrm"
> IFD [Indexer]: delete "_f.prx"
> IFD [Indexer]: delete "_j.tii"
> IFD [Indexer]: delete "_h.tii"
> IFD [Indexer]: delete "_e.prx"
> IFD [Indexer]: delete "_c.frq"
> IFD [Indexer]: delete "_i.prx"
> IFD [Indexer]: delete "_j.fdx"
> IFD [Indexer]: delete "_i.nrm"
> IFD [Indexer]: delete "_j.fdt"
> IFD [Indexer]: delete "_k.prx"
> IFD [Indexer]: delete "_b.nrm"
> IFD [Indexer]: delete "_e.tis"
> IFD [Indexer]: delete "_e.tii"
> IFD [Indexer]: delete "_g.fnm"
> IFD [Indexer]: delete "_k.nrm"
> IFD [Indexer]: delete "_j.nrm"
> IFD [Indexer]: delete "_j.prx"
> IFD [Indexer]: delete "_e.nrm"
> IFD [Indexer]: delete "_g.prx"
> IFD [Indexer]: delete "_b.prx"
> IFD [Indexer]: delete "_i.frq"
> IFD [Indexer]: delete "_b.tis"
> IFD [Indexer]: delete "_d.prx"
> IFD [Indexer]: delete "_g.frq"
> IFD [Indexer]: delete "_d.fdx"
> IFD [Indexer]: delete "_b.tii"
> IFD [Indexer]: delete "_b.frq"
> IFD [Indexer]: delete "_d.fdt"
> IFD [Indexer]: delete "_f.fnm"
> IFD [Indexer]: delete "_j.fnm"
> IFD [Indexer]: delete "_b.fnm"
> IFD [Indexer]: delete "_d.nrm"
> IFD [Indexer]: delete "_b.fdt"
> IFD [Indexer]: delete "_k.frq"
> IFD [Indexer]: delete "_c.fnm"
> IFD [Indexer]: delete "_j.frq"
> IFD [Indexer]: delete "_i.fnm"
> IFD [Indexer]: delete "_f.frq"
> IFD [Indexer]: delete "_b.fdx"
> IW 9 [Indexer]: commit: done
> IW 9 [Indexer]: at close: _a:C9780 _l:C9909
>
> =====================================================
>
> IFD [Indexer]: setInfoStream
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
> IW 10 [Indexer]: setInfoStream:
> dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
> pathname>\lresumes1.search.main.1 autoCommit=false
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB=16.0
> maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=100000 index=_a:C9780 _l:C9909
> IW 10 [Indexer]: flush at getReader
> IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
> IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
> allocMB=15.789 deletesMB=0.212 triggerMB=16
> IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
> docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
> numDocs=1526 numBufDelTerms=1526
> IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
> IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m numDocs=1526
> IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
> newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
> IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> isCommit = false]
> IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
> isCommit = false]
> IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
> IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2 segments
> IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1 segments
> IW 10 [UpdWriterBuild : 10]: CMS: now merge
> IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
> IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now return
> IW 10 [Indexer]: prepareCommit: flush
> IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m docStoreOffset=1526
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
> numBufDelTerms=501
> IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> IW 10 [Indexer]: flush shared docStore segment _m
> IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
> numDocs=2027
> IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
> IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
> docs/MB=316.593 new/old=23.953%
> IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = false]
> IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted docIDs
> and 0 deleted queries on 4 segments.
> IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = false]
> IW 10 [Indexer]: LMP: findMerges: 4 segments
> IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
> IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
> IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
> IW 10 [Indexer]: CMS: now merge
> IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> IW 10 [Indexer]: CMS: no more merges pending; now return
> IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m changeCount=6
> IW 10 [Indexer]: now sync _m.frq
> IW 10 [Indexer]: now sync _m.prx
> IW 10 [Indexer]: now sync _n.tis
> IW 10 [Indexer]: now sync _m.tis
> IW 10 [Indexer]: now sync _n.frq
> IW 10 [Indexer]: now sync _n.tii
> IW 10 [Indexer]: now sync _m.tii
> IW 10 [Indexer]: now sync _m.fnm
> IW 10 [Indexer]: now sync _n.fnm
> IW 10 [Indexer]: now sync _n.nrm
> IW 10 [Indexer]: now sync _n.prx
> IW 10 [Indexer]: now sync _m.nrm
> IW 10 [Indexer]: now sync _m.fdt
> IW 10 [Indexer]: now sync _m.fdx
> IW 10 [Indexer]: done all syncs
> IW 10 [Indexer]: flush at getReader
> IW 10 [Indexer]: flush: segment=null docStoreSegment=_n docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
> numBufDelTerms=0
> IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m
> IW 10 [Indexer]: flush shared docStore segment _n
> IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n numDocs=0
> IW 10 [Indexer]: commit: start
> IW 10 [Indexer]: commit: already prepared
> IW 10 [Indexer]: commit: pendingCommit != null
> IW 10 [Indexer]: commit: wrote segments file "segments_e"
> IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = true]
> IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
> IFD [Indexer]: delete "segments_d"
> IW 10 [Indexer]: commit: done
> IW 10 [Indexer]: flush at getReader
> IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m
> IW 10 [Indexer]: now flush at close
> IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> numBufDelTerms=0
> IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
> _n:C501->_m
> IW 10 [Indexer]: CMS: now merge
> IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
> IW 10 [Indexer]: CMS: no more merges pending; now return
> IW 10 [Indexer]: now call final commit()
> IW 10 [Indexer]: startCommit(): start sizeInBytes=0
> IW 10 [Indexer]: skip startCommit(): no changes pending
> IW 10 [Indexer]: commit: pendingCommit == null; skip
> IW 10 [Indexer]: commit: done
> IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>
>
> Peter
>
> On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Do you know the place in the infoStream output where you got a reader
>> with the wrong (unexplained extra +1) version? If so, can you post
>> the infoStream output up to that point?
>>
>> Mike
>>
>> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <pe...@gmail.com>
>> wrote:
>> > I've reproduced this and I have a bunch of infoStream log files. Since
>> the
>> > messages have no timestamps, it's hard to tell where the relevant entries
>> > are. What should I be looking for?
>> >
>> > Peter
>> >
>> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <peterlkeegan@gmail.com
>> >wrote:
>> >
>> >> I'm pretty sure there are flushes and segment merges going on, but as
>> you
>> >> said, that shouldn't affect the version increment. I'll see what I can
>> do to
>> >> get infoStream output.
>> >>
>> >> Thanks,
>> >> Peter
>> >>
>> >>
>> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
>> >> lucene@mikemccandless.com> wrote:
>> >>
>> >>> Well I'm at a loss then. The version should only increment on commit.
>> >>>
>> >>> Can you make it all happen when infoStream is on, and post back?
>> >>>
>> >>> Mike
>> >>>
>> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <peterlkeegan@gmail.com
>> >
>> >>> wrote:
>> >>> > Only one writer thread and one writer process.
>> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
>> >>> > MaxFieldLength mfl), which sets autocommit=false.
>> >>> >
>> >>> > Peter
>> >>> >
>> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>> >>> > lucene@mikemccandless.com> wrote:
>> >>> >
>> >>> >> That's curious.
>> >>> >>
>> >>> >> It's only on prepareCommit (or, commit, if you didn't first prepare,
>> >>> >> since that will call prepareCommit internally) that this version
>> >>> >> should increase.
>> >>> >>
>> >>> >> Is there only 1 thread doing this?
>> >>> >>
>> >>> >> Oh, and, are you passing false for autoCommit?
>> >>> >>
>> >>> >> Mike
>> >>> >>
>> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
>> peterlkeegan@gmail.com
>> >>> >
>> >>> >> wrote:
>> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
>> >>> repeated
>> >>> >> at
>> >>> >> > regular intervals:
>> >>> >> >
>> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
>> >>> >> maxFieldLen)
>> >>> >> > 4. writer.getReader().getVersion();
>> >>> >> > 5. writer.prepareCommit();
>> >>> >> > 6. writer.getReader().getVersion();
>> >>> >> > 7. writer.commit();
>> >>> >> > 8. writer.close();
>> >>> >> >
>> >>> >> > I'm using the version number to keep external data in synch with
>> the
>> >>> >> index.
>> >>> >> > Usually, the version number from (6) is 1 greater than from (4)
>> and
>> >>> the
>> >>> >> > version from (4) equals the version from the previous (6). At
>> least
>> >>> once
>> >>> >> a
>> >>> >> > day, however, the version from (4) is 1 greater than from the
>> >>> previous
>> >>> >> (6).
>> >>> >> > What would explain this sporadic behavior of version numbers?
>> >>> >> >
>> >>> >> > Thanks,
>> >>> >> > Peter
>> >>> >> >
>> >>> >>
>> >>> >>
>> ---------------------------------------------------------------------
>> >>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>> >>
>> >>> >>
>> >>> >
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>>
>> >>>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
I'm pretty sure this output occurred when the version number skipped +1.
The line containing '============'. separates the close/open IndexWriter.
IFD [Indexer]: setInfoStream
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
IW 9 [Indexer]: setInfoStream:
dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
pathname>\lresumes1.search.main.1 autoCommit=false
mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB=16.0
maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
IW 9 [Indexer]: flush at getReader
IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816
allocMB=15.816 deletesMB=0.203 triggerMB=16
IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j
docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
numDocs=1456 numBufDelTerms=1456
IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b
_c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
_i:C703->_h
IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j numDocs=1456
IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704 newFlushedSize=4969789
docs/MB=307.202 new/old=29.966%
IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
isCommit = false]
IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ;
isCommit = false]
IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 segments
IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 segments
IW 9 [UpdWriterBuild : 9]: CMS: now merge
IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j
IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return
IW 9 [Indexer]: prepareCommit: flush
IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j docStoreOffset=1456
flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
numBufDelTerms=509
IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j
IW 9 [Indexer]: flush shared docStore segment _j
IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j
numDocs=1965
IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970
docs/MB=287.727 new/old=24.788%
IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = false]
IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted docIDs
and 0 deleted queries on 11 segments.
IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = false]
IW 9 [Indexer]: LMP: findMerges: 11 segments
IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments
IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments
IW 9 [Indexer]: LMP: 1 to 11: add this merge
IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j [total 1 pending]
IW 9 [Indexer]: CMS: now merge
IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
_k:C509->_j
IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
_k:C509->_j into _l [mergeDocStores]
IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0]
IW 9 [Indexer]: CMS: no more merges pending; now return
IW 9 [Indexer]: startCommit(): start sizeInBytes=0
IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j changeCount=7
IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start
IW 9 [Indexer]: now sync _k.fnm
IW 9 [Lucene Merge Thread #0]: now merge
merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
_g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l
[mergeDocStores]
merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
_f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
_k:C509->_j into _l [mergeDocStores]
IW 9 [Indexer]: now sync _k.tii
IW 9 [Indexer]: now sync _k.tis
IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
IW 9 [Indexer]: now sync _j.tis
IW 9 [Indexer]: now sync _j.tii
IW 9 [Indexer]: now sync _j.fdx
IW 9 [Indexer]: now sync _j.fdt
IW 9 [Indexer]: now sync _k.prx
IW 9 [Indexer]: now sync _k.nrm
IW 9 [Indexer]: now sync _j.nrm
IW 9 [Indexer]: now sync _j.prx
IW 9 [Indexer]: now sync _j.fnm
IW 9 [Indexer]: now sync _k.frq
IW 9 [Indexer]: now sync _j.frq
IW 9 [Indexer]: done all syncs
IW 9 [Indexer]: flush at getReader
IW 9 [Indexer]: flush: segment=null docStoreSegment=_k docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
numBufDelTerms=0
IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j
IW 9 [Indexer]: flush shared docStore segment _k
IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k numDocs=0
IW 9 [Indexer]: commit: start
IW 9 [Indexer]: commit: already prepared
IW 9 [Indexer]: commit: pendingCommit != null
IW 9 [Indexer]: commit: wrote segments file "segments_c"
IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit = true]
IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
IFD [Indexer]: delete "segments_b"
IW 9 [Indexer]: commit: done
IW 9 [Indexer]: flush at getReader
IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j
IW 9 [Indexer]: now flush at close
IW 9 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j
IW 9 [Indexer]: CMS: now merge
IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
_k:C509->_j
IW 9 [Indexer]: CMS: no more merges pending; now return
IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
_h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b _c:C717->_b
_d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
_j:C1456->_j _k:C509->_j into _l [mergeDocStores]
IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
isCommit = false]
IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments ;
isCommit = false]
IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done
IW 9 [Indexer]: now call final commit()
IW 9 [Indexer]: startCommit(): start sizeInBytes=0
IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
IW 9 [Indexer]: now sync _l.nrm
IW 9 [Indexer]: now sync _l.fnm
IW 9 [Indexer]: now sync _l.prx
IW 9 [Indexer]: now sync _l.frq
IW 9 [Indexer]: now sync _l.tis
IW 9 [Indexer]: now sync _l.fdx
IW 9 [Indexer]: now sync _l.fdt
IW 9 [Indexer]: now sync _l.tii
IW 9 [Indexer]: done all syncs
IW 9 [Indexer]: commit: pendingCommit != null
IW 9 [Indexer]: commit: wrote segments file "segments_d"
IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = true]
IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
IFD [Indexer]: delete "_h.prx"
IFD [Indexer]: delete "_f.tis"
IFD [Indexer]: delete "_g.tis"
IFD [Indexer]: delete "_d.tii"
IFD [Indexer]: delete "_e.frq"
IFD [Indexer]: delete "_k.fnm"
IFD [Indexer]: delete "_c.tii"
IFD [Indexer]: delete "_h.nrm"
IFD [Indexer]: delete "_i.tis"
IFD [Indexer]: delete "_k.tii"
IFD [Indexer]: delete "_h.fdt"
IFD [Indexer]: delete "_c.tis"
IFD [Indexer]: delete "_g.tii"
IFD [Indexer]: delete "_h.fdx"
IFD [Indexer]: delete "_c.nrm"
IFD [Indexer]: delete "_i.tii"
IFD [Indexer]: delete "_d.tis"
IFD [Indexer]: delete "_f.tii"
IFD [Indexer]: delete "_e.fnm"
IFD [Indexer]: delete "_k.tis"
IFD [Indexer]: delete "_d.frq"
IFD [Indexer]: delete "_f.nrm"
IFD [Indexer]: delete "_h.fnm"
IFD [Indexer]: delete "segments_c"
IFD [Indexer]: delete "_c.prx"
IFD [Indexer]: delete "_d.fnm"
IFD [Indexer]: delete "_h.frq"
IFD [Indexer]: delete "_h.tis"
IFD [Indexer]: delete "_f.fdt"
IFD [Indexer]: delete "_j.tis"
IFD [Indexer]: delete "_f.fdx"
IFD [Indexer]: delete "_g.nrm"
IFD [Indexer]: delete "_f.prx"
IFD [Indexer]: delete "_j.tii"
IFD [Indexer]: delete "_h.tii"
IFD [Indexer]: delete "_e.prx"
IFD [Indexer]: delete "_c.frq"
IFD [Indexer]: delete "_i.prx"
IFD [Indexer]: delete "_j.fdx"
IFD [Indexer]: delete "_i.nrm"
IFD [Indexer]: delete "_j.fdt"
IFD [Indexer]: delete "_k.prx"
IFD [Indexer]: delete "_b.nrm"
IFD [Indexer]: delete "_e.tis"
IFD [Indexer]: delete "_e.tii"
IFD [Indexer]: delete "_g.fnm"
IFD [Indexer]: delete "_k.nrm"
IFD [Indexer]: delete "_j.nrm"
IFD [Indexer]: delete "_j.prx"
IFD [Indexer]: delete "_e.nrm"
IFD [Indexer]: delete "_g.prx"
IFD [Indexer]: delete "_b.prx"
IFD [Indexer]: delete "_i.frq"
IFD [Indexer]: delete "_b.tis"
IFD [Indexer]: delete "_d.prx"
IFD [Indexer]: delete "_g.frq"
IFD [Indexer]: delete "_d.fdx"
IFD [Indexer]: delete "_b.tii"
IFD [Indexer]: delete "_b.frq"
IFD [Indexer]: delete "_d.fdt"
IFD [Indexer]: delete "_f.fnm"
IFD [Indexer]: delete "_j.fnm"
IFD [Indexer]: delete "_b.fnm"
IFD [Indexer]: delete "_d.nrm"
IFD [Indexer]: delete "_b.fdt"
IFD [Indexer]: delete "_k.frq"
IFD [Indexer]: delete "_c.fnm"
IFD [Indexer]: delete "_j.frq"
IFD [Indexer]: delete "_i.fnm"
IFD [Indexer]: delete "_f.frq"
IFD [Indexer]: delete "_b.fdx"
IW 9 [Indexer]: commit: done
IW 9 [Indexer]: at close: _a:C9780 _l:C9909
=====================================================
IFD [Indexer]: setInfoStream
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
IW 10 [Indexer]: setInfoStream:
dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
pathname>\lresumes1.search.main.1 autoCommit=false
mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB=16.0
maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
maxFieldLength=100000 index=_a:C9780 _l:C9909
IW 10 [Indexer]: flush at getReader
IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 10 [Indexer]: index before flush _a:C9780 _l:C9909
IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789
allocMB=15.789 deletesMB=0.212 triggerMB=16
IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m
docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
numDocs=1526 numBufDelTerms=1526
IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909
IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m numDocs=1526
IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032
newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
isCommit = false]
IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ;
isCommit = false]
IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2 segments
IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1 segments
IW 10 [UpdWriterBuild : 10]: CMS: now merge
IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m
IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now return
IW 10 [Indexer]: prepareCommit: flush
IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m docStoreOffset=1526
flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
numBufDelTerms=501
IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
IW 10 [Indexer]: flush shared docStore segment _m
IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m
numDocs=2027
IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342
docs/MB=316.593 new/old=23.953%
IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = false]
IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted docIDs
and 0 deleted queries on 4 segments.
IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = false]
IW 10 [Indexer]: LMP: findMerges: 4 segments
IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments
IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments
IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments
IW 10 [Indexer]: CMS: now merge
IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
IW 10 [Indexer]: CMS: no more merges pending; now return
IW 10 [Indexer]: startCommit(): start sizeInBytes=0
IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
_n:C501->_m changeCount=6
IW 10 [Indexer]: now sync _m.frq
IW 10 [Indexer]: now sync _m.prx
IW 10 [Indexer]: now sync _n.tis
IW 10 [Indexer]: now sync _m.tis
IW 10 [Indexer]: now sync _n.frq
IW 10 [Indexer]: now sync _n.tii
IW 10 [Indexer]: now sync _m.tii
IW 10 [Indexer]: now sync _m.fnm
IW 10 [Indexer]: now sync _n.fnm
IW 10 [Indexer]: now sync _n.nrm
IW 10 [Indexer]: now sync _n.prx
IW 10 [Indexer]: now sync _m.nrm
IW 10 [Indexer]: now sync _m.fdt
IW 10 [Indexer]: now sync _m.fdx
IW 10 [Indexer]: done all syncs
IW 10 [Indexer]: flush at getReader
IW 10 [Indexer]: flush: segment=null docStoreSegment=_n docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
numBufDelTerms=0
IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
_n:C501->_m
IW 10 [Indexer]: flush shared docStore segment _n
IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n numDocs=0
IW 10 [Indexer]: commit: start
IW 10 [Indexer]: commit: already prepared
IW 10 [Indexer]: commit: pendingCommit != null
IW 10 [Indexer]: commit: wrote segments file "segments_e"
IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = true]
IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
IFD [Indexer]: delete "segments_d"
IW 10 [Indexer]: commit: done
IW 10 [Indexer]: flush at getReader
IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
_n:C501->_m
IW 10 [Indexer]: now flush at close
IW 10 [Indexer]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
numBufDelTerms=0
IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m
_n:C501->_m
IW 10 [Indexer]: CMS: now merge
IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
IW 10 [Indexer]: CMS: no more merges pending; now return
IW 10 [Indexer]: now call final commit()
IW 10 [Indexer]: startCommit(): start sizeInBytes=0
IW 10 [Indexer]: skip startCommit(): no changes pending
IW 10 [Indexer]: commit: pendingCommit == null; skip
IW 10 [Indexer]: commit: done
IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
Peter
On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Do you know the place in the infoStream output where you got a reader
> with the wrong (unexplained extra +1) version? If so, can you post
> the infoStream output up to that point?
>
> Mike
>
> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <pe...@gmail.com>
> wrote:
> > I've reproduced this and I have a bunch of infoStream log files. Since
> the
> > messages have no timestamps, it's hard to tell where the relevant entries
> > are. What should I be looking for?
> >
> > Peter
> >
> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <peterlkeegan@gmail.com
> >wrote:
> >
> >> I'm pretty sure there are flushes and segment merges going on, but as
> you
> >> said, that shouldn't affect the version increment. I'll see what I can
> do to
> >> get infoStream output.
> >>
> >> Thanks,
> >> Peter
> >>
> >>
> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
> >> lucene@mikemccandless.com> wrote:
> >>
> >>> Well I'm at a loss then. The version should only increment on commit.
> >>>
> >>> Can you make it all happen when infoStream is on, and post back?
> >>>
> >>> Mike
> >>>
> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <peterlkeegan@gmail.com
> >
> >>> wrote:
> >>> > Only one writer thread and one writer process.
> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
> >>> > MaxFieldLength mfl), which sets autocommit=false.
> >>> >
> >>> > Peter
> >>> >
> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
> >>> > lucene@mikemccandless.com> wrote:
> >>> >
> >>> >> That's curious.
> >>> >>
> >>> >> It's only on prepareCommit (or, commit, if you didn't first prepare,
> >>> >> since that will call prepareCommit internally) that this version
> >>> >> should increase.
> >>> >>
> >>> >> Is there only 1 thread doing this?
> >>> >>
> >>> >> Oh, and, are you passing false for autoCommit?
> >>> >>
> >>> >> Mike
> >>> >>
> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <
> peterlkeegan@gmail.com
> >>> >
> >>> >> wrote:
> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
> >>> repeated
> >>> >> at
> >>> >> > regular intervals:
> >>> >> >
> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
> >>> >> maxFieldLen)
> >>> >> > 4. writer.getReader().getVersion();
> >>> >> > 5. writer.prepareCommit();
> >>> >> > 6. writer.getReader().getVersion();
> >>> >> > 7. writer.commit();
> >>> >> > 8. writer.close();
> >>> >> >
> >>> >> > I'm using the version number to keep external data in synch with
> the
> >>> >> index.
> >>> >> > Usually, the version number from (6) is 1 greater than from (4)
> and
> >>> the
> >>> >> > version from (4) equals the version from the previous (6). At
> least
> >>> once
> >>> >> a
> >>> >> > day, however, the version from (4) is 1 greater than from the
> >>> previous
> >>> >> (6).
> >>> >> > What would explain this sporadic behavior of version numbers?
> >>> >> >
> >>> >> > Thanks,
> >>> >> > Peter
> >>> >> >
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>> >>
> >>> >>
> >>> >
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
Do you know the place in the infoStream output where you got a reader
with the wrong (unexplained extra +1) version? If so, can you post
the infoStream output up to that point?
Mike
On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <pe...@gmail.com> wrote:
> I've reproduced this and I have a bunch of infoStream log files. Since the
> messages have no timestamps, it's hard to tell where the relevant entries
> are. What should I be looking for?
>
> Peter
>
> On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <pe...@gmail.com>wrote:
>
>> I'm pretty sure there are flushes and segment merges going on, but as you
>> said, that shouldn't affect the version increment. I'll see what I can do to
>> get infoStream output.
>>
>> Thanks,
>> Peter
>>
>>
>> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>> Well I'm at a loss then. The version should only increment on commit.
>>>
>>> Can you make it all happen when infoStream is on, and post back?
>>>
>>> Mike
>>>
>>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <pe...@gmail.com>
>>> wrote:
>>> > Only one writer thread and one writer process.
>>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
>>> > MaxFieldLength mfl), which sets autocommit=false.
>>> >
>>> > Peter
>>> >
>>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>>> > lucene@mikemccandless.com> wrote:
>>> >
>>> >> That's curious.
>>> >>
>>> >> It's only on prepareCommit (or, commit, if you didn't first prepare,
>>> >> since that will call prepareCommit internally) that this version
>>> >> should increase.
>>> >>
>>> >> Is there only 1 thread doing this?
>>> >>
>>> >> Oh, and, are you passing false for autoCommit?
>>> >>
>>> >> Mike
>>> >>
>>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <peterlkeegan@gmail.com
>>> >
>>> >> wrote:
>>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
>>> repeated
>>> >> at
>>> >> > regular intervals:
>>> >> >
>>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
>>> >> maxFieldLen)
>>> >> > 4. writer.getReader().getVersion();
>>> >> > 5. writer.prepareCommit();
>>> >> > 6. writer.getReader().getVersion();
>>> >> > 7. writer.commit();
>>> >> > 8. writer.close();
>>> >> >
>>> >> > I'm using the version number to keep external data in synch with the
>>> >> index.
>>> >> > Usually, the version number from (6) is 1 greater than from (4) and
>>> the
>>> >> > version from (4) equals the version from the previous (6). At least
>>> once
>>> >> a
>>> >> > day, however, the version from (4) is 1 greater than from the
>>> previous
>>> >> (6).
>>> >> > What would explain this sporadic behavior of version numbers?
>>> >> >
>>> >> > Thanks,
>>> >> > Peter
>>> >> >
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
I've reproduced this and I have a bunch of infoStream log files. Since the
messages have no timestamps, it's hard to tell where the relevant entries
are. What should I be looking for?
Peter
On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <pe...@gmail.com>wrote:
> I'm pretty sure there are flushes and segment merges going on, but as you
> said, that shouldn't affect the version increment. I'll see what I can do to
> get infoStream output.
>
> Thanks,
> Peter
>
>
> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Well I'm at a loss then. The version should only increment on commit.
>>
>> Can you make it all happen when infoStream is on, and post back?
>>
>> Mike
>>
>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <pe...@gmail.com>
>> wrote:
>> > Only one writer thread and one writer process.
>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
>> > MaxFieldLength mfl), which sets autocommit=false.
>> >
>> > Peter
>> >
>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
>> > lucene@mikemccandless.com> wrote:
>> >
>> >> That's curious.
>> >>
>> >> It's only on prepareCommit (or, commit, if you didn't first prepare,
>> >> since that will call prepareCommit internally) that this version
>> >> should increase.
>> >>
>> >> Is there only 1 thread doing this?
>> >>
>> >> Oh, and, are you passing false for autoCommit?
>> >>
>> >> Mike
>> >>
>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <peterlkeegan@gmail.com
>> >
>> >> wrote:
>> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
>> repeated
>> >> at
>> >> > regular intervals:
>> >> >
>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
>> >> maxFieldLen)
>> >> > 4. writer.getReader().getVersion();
>> >> > 5. writer.prepareCommit();
>> >> > 6. writer.getReader().getVersion();
>> >> > 7. writer.commit();
>> >> > 8. writer.close();
>> >> >
>> >> > I'm using the version number to keep external data in synch with the
>> >> index.
>> >> > Usually, the version number from (6) is 1 greater than from (4) and
>> the
>> >> > version from (4) equals the version from the previous (6). At least
>> once
>> >> a
>> >> > day, however, the version from (4) is 1 greater than from the
>> previous
>> >> (6).
>> >> > What would explain this sporadic behavior of version numbers?
>> >> >
>> >> > Thanks,
>> >> > Peter
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
I'm pretty sure there are flushes and segment merges going on, but as you
said, that shouldn't affect the version increment. I'll see what I can do to
get infoStream output.
Thanks,
Peter
On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Well I'm at a loss then. The version should only increment on commit.
>
> Can you make it all happen when infoStream is on, and post back?
>
> Mike
>
> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <pe...@gmail.com>
> wrote:
> > Only one writer thread and one writer process.
> > I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
> > MaxFieldLength mfl), which sets autocommit=false.
> >
> > Peter
> >
> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> That's curious.
> >>
> >> It's only on prepareCommit (or, commit, if you didn't first prepare,
> >> since that will call prepareCommit internally) that this version
> >> should increase.
> >>
> >> Is there only 1 thread doing this?
> >>
> >> Oh, and, are you passing false for autoCommit?
> >>
> >> Mike
> >>
> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <pe...@gmail.com>
> >> wrote:
> >> > Using Lucene 2.9.1, I have the following pseudocode which gets
> repeated
> >> at
> >> > regular intervals:
> >> >
> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
> >> maxFieldLen)
> >> > 4. writer.getReader().getVersion();
> >> > 5. writer.prepareCommit();
> >> > 6. writer.getReader().getVersion();
> >> > 7. writer.commit();
> >> > 8. writer.close();
> >> >
> >> > I'm using the version number to keep external data in synch with the
> >> index.
> >> > Usually, the version number from (6) is 1 greater than from (4) and
> the
> >> > version from (4) equals the version from the previous (6). At least
> once
> >> a
> >> > day, however, the version from (4) is 1 greater than from the previous
> >> (6).
> >> > What would explain this sporadic behavior of version numbers?
> >> >
> >> > Thanks,
> >> > Peter
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
Well I'm at a loss then. The version should only increment on commit.
Can you make it all happen when infoStream is on, and post back?
Mike
On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan <pe...@gmail.com> wrote:
> Only one writer thread and one writer process.
> I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
> MaxFieldLength mfl), which sets autocommit=false.
>
> Peter
>
> On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> That's curious.
>>
>> It's only on prepareCommit (or, commit, if you didn't first prepare,
>> since that will call prepareCommit internally) that this version
>> should increase.
>>
>> Is there only 1 thread doing this?
>>
>> Oh, and, are you passing false for autoCommit?
>>
>> Mike
>>
>> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <pe...@gmail.com>
>> wrote:
>> > Using Lucene 2.9.1, I have the following pseudocode which gets repeated
>> at
>> > regular intervals:
>> >
>> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
>> maxFieldLen)
>> > 4. writer.getReader().getVersion();
>> > 5. writer.prepareCommit();
>> > 6. writer.getReader().getVersion();
>> > 7. writer.commit();
>> > 8. writer.close();
>> >
>> > I'm using the version number to keep external data in synch with the
>> index.
>> > Usually, the version number from (6) is 1 greater than from (4) and the
>> > version from (4) equals the version from the previous (6). At least once
>> a
>> > day, however, the version from (4) is 1 greater than from the previous
>> (6).
>> > What would explain this sporadic behavior of version numbers?
>> >
>> > Thanks,
>> > Peter
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Peter Keegan <pe...@gmail.com>.
Only one writer thread and one writer process.
I'm calling IndexWriter(Directory d, Analyzer a, boolean create,
MaxFieldLength mfl), which sets autocommit=false.
Peter
On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> That's curious.
>
> It's only on prepareCommit (or, commit, if you didn't first prepare,
> since that will call prepareCommit internally) that this version
> should increase.
>
> Is there only 1 thread doing this?
>
> Oh, and, are you passing false for autoCommit?
>
> Mike
>
> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <pe...@gmail.com>
> wrote:
> > Using Lucene 2.9.1, I have the following pseudocode which gets repeated
> at
> > regular intervals:
> >
> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false,
> maxFieldLen)
> > 4. writer.getReader().getVersion();
> > 5. writer.prepareCommit();
> > 6. writer.getReader().getVersion();
> > 7. writer.commit();
> > 8. writer.close();
> >
> > I'm using the version number to keep external data in synch with the
> index.
> > Usually, the version number from (6) is 1 greater than from (4) and the
> > version from (4) equals the version from the previous (6). At least once
> a
> > day, however, the version from (4) is 1 greater than from the previous
> (6).
> > What would explain this sporadic behavior of version numbers?
> >
> > Thanks,
> > Peter
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter.getReader.getVersion behavior
Posted by Michael McCandless <lu...@mikemccandless.com>.
That's curious.
It's only on prepareCommit (or, commit, if you didn't first prepare,
since that will call prepareCommit internally) that this version
should increase.
Is there only 1 thread doing this?
Oh, and, are you passing false for autoCommit?
Mike
On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan <pe...@gmail.com> wrote:
> Using Lucene 2.9.1, I have the following pseudocode which gets repeated at
> regular intervals:
>
> 1. FSDirectory dir = FSDirectory.open(java.io.File);
> 2. dir.setLockFactory(new SingleInstanceLockFactory());
> 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false, maxFieldLen)
> 4. writer.getReader().getVersion();
> 5. writer.prepareCommit();
> 6. writer.getReader().getVersion();
> 7. writer.commit();
> 8. writer.close();
>
> I'm using the version number to keep external data in synch with the index.
> Usually, the version number from (6) is 1 greater than from (4) and the
> version from (4) equals the version from the previous (6). At least once a
> day, however, the version from (4) is 1 greater than from the previous (6).
> What would explain this sporadic behavior of version numbers?
>
> Thanks,
> Peter
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.getReader.getVersion behavior
Posted by Jason Rutherglen <ja...@gmail.com>.
Peter,
Perhaps other concurrent operations?
Jason
On Tue, Feb 23, 2010 at 10:43 AM, Peter Keegan <pe...@gmail.com> wrote:
> Using Lucene 2.9.1, I have the following pseudocode which gets repeated at
> regular intervals:
>
> 1. FSDirectory dir = FSDirectory.open(java.io.File);
> 2. dir.setLockFactory(new SingleInstanceLockFactory());
> 3. IndexWriter writer = new IndexWriter(dir, Analyzer, false, maxFieldLen)
> 4. writer.getReader().getVersion();
> 5. writer.prepareCommit();
> 6. writer.getReader().getVersion();
> 7. writer.commit();
> 8. writer.close();
>
> I'm using the version number to keep external data in synch with the index.
> Usually, the version number from (6) is 1 greater than from (4) and the
> version from (4) equals the version from the previous (6). At least once a
> day, however, the version from (4) is 1 greater than from the previous (6).
> What would explain this sporadic behavior of version numbers?
>
> Thanks,
> Peter
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org