You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bill Janssen <ja...@parc.com> on 2007/11/28 19:22:54 UTC
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Here's the code I'm using:
try {
// Now add the documents to the index
IndexWriter writer = new IndexWriter(index_loc, new StandardAnalyzer(), !index_loc.exists());
writer.setMaxFieldLength(Integer.MAX_VALUE);
try {
for (int i = start; i < ids.length; i ++) {
docit = build_document_iterator(doc_root_dir, ids[i]);
int count = 0;
while (docit.hasNext()) {
writer.addDocument((Document)(docit.next()));
count += 1;
}
System.out.println("Added " + docit.id + " (" + count + " versions)");
System.out.flush();
}
} finally {
// And close the index
System.out.println("Optimizing...");
writer.optimize();
writer.close();
}
} catch (Exception e) {
e.printStackTrace(System.err);
}
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> I'm going to run the same software on an
> Intel machine and see what happens.
So, I ran the same codebase with lucene-core-2.2.0.jar on an Intel Mac
Pro, OS X 10.5.0, Java 1.5, and no exception is raised. Different
corpus, about 50000 pages instead of 20000. This is reinforcing my
thinking that it's a big-endian issue.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> You are not hitting any other exception before this one right?
>
> Can you change your test case so that the "catch" clause is run
> before the "finally" clause? I wonder if you are hitting some
> interesting exception and then trying to optimize, which then
> masks the original exception.
Yes, I thought of that. But when I comment out the call to
"optimize()", no exception is signalled. So I think it's unlikely.
I'm thinking it might be an endianness issue. Do you folks test a lot
on big-endian platforms? Just the kind of thing that affects
bit-string manipulation. I'm going to run the same software on an
Intel machine and see what happens.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
Hey Bill,
Any status on this?
On Dec 2, 2007, at 10:37 PM, Bill Janssen wrote:
>>> Hmmm, it still sounds like you are hitting a threading issue that is
>>> probably exacerbated by the multicore platform of the newer machine.
>>
>> Exactly what I was thinking.
>> What are the details of the CPUs of these two systems?
>
> Ah, good point. The bad machine is a dual-processor 1GHz G4 "wind
> tunnel" model, and the good machine is a 1.5GH single-processor G4
> Powerbook. I'm running the re-indexing in a daemon thread in a server
> which manages the repository.
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> > Hmmm, it still sounds like you are hitting a threading issue that is
> > probably exacerbated by the multicore platform of the newer machine.
>
> Exactly what I was thinking.
> What are the details of the CPUs of these two systems?
Ah, good point. The bad machine is a dual-processor 1GHz G4 "wind
tunnel" model, and the good machine is a 1.5GH single-processor G4
Powerbook. I'm running the re-indexing in a daemon thread in a server
which manages the repository.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 2, 2007 9:28 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Hmmm, it still sounds like you are hitting a threading issue that is
> probably exacerbated by the multicore platform of the newer machine.
Exactly what I was thinking.
What are the details of the CPUs of these two systems?
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> > Another thing to try is turning on the infoStream
> > (IndexWriter.setInfoStream(...)) and capture & post the resulting log.
> > It will be very large since it takes quite a while for the error to
> > occur...
>
> I can do that.
Here's a more complete dump. I've modified the code so that I now
remove any existing versions of the document before re-indexing it and
its pages.
Bill
/Library/Java/Home/bin/java '-Dcom.parc.uplib.indexing.debugMode=true' '-Dcom.parc.uplib.indexing.indexProperties=contents:title:categories$,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-message-id*:email-guid*:email-subject:email-from-name:email-from-address*:email-attachment-to*:email-thread-index*:email-references$,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*' -classpath "/local/uplib/share/UpLib-1.7/code/lucene-core-2.2.0.jar:/local/uplib/share/UpLib-1.7/code/LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000 com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index" update /local/janssen-uplib/docs 01174-15-2815-270 01174-15-2552-042 01173-98-5675-575 01173-98-4457-188 01173-83-8266-533 01173-80-8759-205
updating
doc_root_dir is /local/janssen-uplib/docs
Working on document /local/janssen-uplib/docs/01174-15-2815-270
Adding header 'apparent-mime-type' I to 01174-15-2815-270
Adding header 'authors' IT to 01174-15-2815-270
Adding header 'categories' I (article) to 01174-15-2815-270
Adding header 'date' I (20070317) to 01174-15-2815-270
Adding header 'sha-hash' I to 01174-15-2815-270
Created empty doc Document<stored/uncompressed,indexed<id:01174-15-2815-270> stored/uncompressed,indexed<uplibdate:20070317> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3566): human nature Full-Mental Nudit
page 1 (3100): I know what you're thinking: W
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01174-15-2815-270 (3 versions)
Working on document /local/janssen-uplib/docs/01174-15-2552-042
Adding header 'abstract' IT to 01174-15-2552-042
Adding header 'apparent-mime-type' I to 01174-15-2552-042
Adding header 'categories' I (photo) to 01174-15-2552-042
Adding header 'date' I (20070316) to 01174-15-2552-042
Adding header 'sha-hash' I to 01174-15-2552-042
Created empty doc Document<stored/uncompressed,indexed<id:01174-15-2552-042> stored/uncompressed,indexed<uplibdate:20070317> stored/uncompressed,indexed<uplibtype:whole>>
Added 01174-15-2552-042 (1 versions)
Working on document /local/janssen-uplib/docs/01173-98-5675-575
Adding header 'apparent-mime-type' I to 01173-98-5675-575
Adding header 'authors' IT to 01173-98-5675-575
Adding header 'categories' I (article) to 01173-98-5675-575
Adding header 'categories' I (medical) to 01173-98-5675-575
Adding header 'date' I (20070313) to 01173-98-5675-575
Adding header 'sha-hash' I to 01173-98-5675-575
Created empty doc Document<stored/uncompressed,indexed<id:01173-98-5675-575> stored/uncompressed,indexed<uplibdate:20070315> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (2730): March 13, 2007 DOW JONES REPRI
page 1 (4445): But just how far -- and how fa
page 2 (2638): "We don't sell snow tires," sa
page 3 (981): A spokeswoman for Rite Aid say
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01173-98-5675-575 (5 versions)
Working on document /local/janssen-uplib/docs/01173-98-4457-188
Adding header 'apparent-mime-type' I to 01173-98-4457-188
Adding header 'authors' IT to 01173-98-4457-188
Adding header 'categories' I (article) to 01173-98-4457-188
Adding header 'date' I (19911006) to 01173-98-4457-188
Adding header 'sha-hash' I to 01173-98-4457-188
Created empty doc Document<stored/uncompressed,indexed<id:01173-98-4457-188> stored/uncompressed,indexed<uplibdate:20070315> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (2897): The Economics of the Colonial
merging segments _ram_0 (1 docs) _ram_1 (1 docs) _ram_2 (1 docs) _ram_3 (1 docs) _ram_4 (1 docs) _ram_5 (1 docs) _ram_6 (1 docs) _ram_7 (1 docs) _ram_8 (1 docs) _ram_9 (1 docs) into _1v9 (10 docs)
flush 6 buffered deleted terms on 6 segments.
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3qj" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.fnm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.fdx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.fdt": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.tii": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.tis": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.frq": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.prx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.nrm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3qi"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3qi": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3qi"
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3qk" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.cfs": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3qj"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.fnm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.fdx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.fdt": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.tii": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.tis": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.tis"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.frq": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.prx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.nrm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1v9.nrm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3qj": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3qj"
page 1 (3372): Newsweek. I rely on The Econom
page 2 (3395): print, �I rely on the Wall Str
page 3 (3032): British condescension toward t
page 4 (1100): The �that being so� style of d
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01173-98-4457-188 (6 versions)
Working on document /local/janssen-uplib/docs/01173-83-8266-533
Adding header 'apparent-mime-type' I to 01173-83-8266-533
Adding header 'categories' I (article) to 01173-83-8266-533
Adding header 'categories' I (medical) to 01173-83-8266-533
Adding header 'categories' I (medical/aging) to 01173-83-8266-533
Adding header 'categories' I (medical) to 01173-83-8266-533
Adding header 'categories' I (medical/exercise) to 01173-83-8266-533
Adding header 'categories' I (medical) to 01173-83-8266-533
Adding header 'date' I (20070313) to 01173-83-8266-533
Adding header 'sha-hash' I to 01173-83-8266-533
Adding header 'source' IT to 01173-83-8266-533
Adding header 'title' IT (Study shows why exercise boosts brainpower) to 01173-83-8266-533
Created empty doc Document<stored/uncompressed,indexed<id:01173-83-8266-533> stored/uncompressed,indexed<uplibdate:20070313> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (2282): Powered by SAVE THIS | EMAIL T
page 1 (1040): Exercise generated blood flow
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01173-83-8266-533 (3 versions)
Working on document /local/janssen-uplib/docs/01173-80-8759-205
Adding header 'apparent-mime-type' I to 01173-80-8759-205
Adding header 'categories' I (photo) to 01173-80-8759-205
Adding header 'categories' I (apple) to 01173-80-8759-205
Adding header 'categories' I (apple) to 01173-80-8759-205
Adding header 'date' I (20070312) to 01173-80-8759-205
Adding header 'sha-hash' I to 01173-80-8759-205
Adding header 'title' IT (Apple Store New York showing iPhone ad on March 12th, 2007) to 01173-80-8759-205
Created empty doc Document<stored/uncompressed,indexed<id:01173-80-8759-205> stored/uncompressed,indexed<uplibdate:20070313> stored/uncompressed,indexed<uplibtype:whole>>
Added 01173-80-8759-205 (1 versions)
Optimizing...
merging segments _ram_a (1 docs) _ram_b (1 docs) _ram_c (1 docs) _ram_d (1 docs) _ram_e (1 docs) _ram_f (1 docs) _ram_g (1 docs) _ram_h (1 docs) _ram_i (1 docs) into _1va (9 docs)
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3ql" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fnm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fdx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fdt": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.tii": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.tis": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.frq": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.prx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.nrm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3qk"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3qk": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3qk"
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3qm" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.cfs": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3ql"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fnm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fdx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fdt": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.tii": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.tis": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.tis"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.frq": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.prx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.nrm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.nrm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3ql": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3ql"
merging segments _1v4 (19680 docs) _1v5 (10 docs) _1v6 (9 docs) _1v7 (10 docs) _1v8 (9 docs) _1v9 (10 docs) _1va (9 docs)org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.tis"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.tis"
java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 23754
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:361)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:325)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:297)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:261)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:1883)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1231)
at com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:419)
at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:664)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> > Another thing to try is turning on the infoStream
> > (IndexWriter.setInfoStream(...)) and capture & post the resulting log.
> > It will be very large since it takes quite a while for the error to
> > occur...
>
> I can do that.
Here's what I see:
Optimizing...
merging segments _ram_a (1 docs) _ram_b (1 docs) _ram_c (1 docs) _ram_d (1 docs) _ram_e (1 docs) _ram_f (1 docs) _ram_g (1 docs) _ram_h (1 docs) _ram_i (1 docs) into _1va (9 docs)
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3ql" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fnm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fdx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.fdt": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.tii": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.tis": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.frq": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.prx": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.nrm": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3qk"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3qk": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3qk"
org.apache.lucene.index.IndexFileDeleter@829e6f main: now checkpoint "segments_3qm" [isCommit = true]
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v4_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v5_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v6_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v7_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v8_1.del": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1v9.cfs": pre-incr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: IncRef "_1va.cfs": pre-incr count is 0
org.apache.lucene.index.IndexFileDeleter@829e6f main: deleteCommits: now remove commit "segments_3ql"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v4_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v5_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v6_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v7_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v8_1.del": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1v9.cfs": pre-decr count is 2
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fnm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fdx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.fdt": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.tii": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.tis": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.tis"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.frq": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.prx": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "_1va.nrm": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1va.nrm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: DecRef "segments_3ql": pre-decr count is 1
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "segments_3ql"
merging segments _1v4 (19680 docs) _1v5 (10 docs) _1v6 (9 docs) _1v7 (10 docs) _1v8 (9 docs) _1v9 (10 docs) _1va (9 docs)org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fdt"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fdx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.fnm"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.frq"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.prx"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.tii"
org.apache.lucene.index.IndexFileDeleter@829e6f main: refresh: removing newly created unreferenced file "_1vb.tis"
org.apache.lucene.index.IndexFileDeleter@829e6f main: delete "_1vb.tis"
java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 23754
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:361)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:325)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:297)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:261)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:1883)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1231)
at com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:419)
at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:664)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Do you have another PPC machine to reproduce this on? (To rule out
> bad RAM/hard-drive on the first one).
I'll dig up an old laptop and try it there.
> Another thing to try is turning on the infoStream
> (IndexWriter.setInfoStream(...)) and capture & post the resulting log.
> It will be very large since it takes quite a while for the error to
> occur...
I can do that.
> Lucene is endian independent: all writes to files boil eventually down
> to a writeByte/writeBytes calls in o.a.l.store.IndexOutput such that
> the ordering is controlled by Lucene, not the underlying CPU
> architecture.
I was actually thinking about the implementation of the bitstrings,
rather than data storage proper.
> Are you using term vectors,
> stored fields, payloads on any of these?
Stored fields. I store a document ID (a 23-character string) for each.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
Hmmm, it still sounds like you are hitting a threading issue that is
probably exacerbated by the multicore platform of the newer machine.
Is there anyway to put together a unit test that we can try?
Thanks,
Grant
On Dec 2, 2007, at 9:10 PM, Bill Janssen wrote:
>> I'll see if I can get back to this over the weekend.
>
> I got a chance to copy my corpus to another G4 and try indexing with
> Lucene 2.2. This one seems OK! Same texts. So now I'm inclined to
> believe that it *is* the machine, rather than the code. Whew! Though
> that doesn't explain why 2.0 works fine on the same corpus on the same
> machine, in repeated tests.
>
> I'm going to run some diagnostics on the first G4, and see if I can
> find a hardware problem. I'll report back.
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> I'll see if I can get back to this over the weekend.
I got a chance to copy my corpus to another G4 and try indexing with
Lucene 2.2. This one seems OK! Same texts. So now I'm inclined to
believe that it *is* the machine, rather than the code. Whew! Though
that doesn't explain why 2.0 works fine on the same corpus on the same
machine, in repeated tests.
I'm going to run some diagnostics on the first G4, and see if I can
find a hardware problem. I'll report back.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Your errors seem to happen around the same area (~20K docs). If you
> skip the first say ~18K docs does the error still happen? We need to
> somehow narrow this down.
I'm trying to boil down the documents to a set which I can deploy on a
DVD-ROM, so I can move the same set around from machine to machine. But
today, unfortunately, I have to do some real work :-).
I'll see if I can get back to this over the weekend.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
I have PPC and Intel access if that helps. Just need a test case.
On Nov 29, 2007, at 5:37 PM, Michael McCandless wrote:
>
> "Bill Janssen" <ja...@parc.com> wrote:
>> No. It's in another location, but perhaps I can get it tomorrow.
>> On the other hand, the success when using 2.0 makes it likely to me
>> that the machine isn't the problem.
>
> Yeah good point. Seems like a long shot (wishful thinking on my
> part!).
>
> Your errors seem to happen around the same area (~20K docs). If you
> skip the first say ~18K docs does the error still happen? We need to
> somehow narrow this down.
>
> Or is there any way I could get a temporary account to log into this
> box and try to track this down? (If indeed it doesn't happen on an
> x86 box -- I unfortunately don't have access to a PPC machine).
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Bill Janssen" <ja...@parc.com> wrote:
> No. It's in another location, but perhaps I can get it tomorrow.
> On the other hand, the success when using 2.0 makes it likely to me
> that the machine isn't the problem.
Yeah good point. Seems like a long shot (wishful thinking on my
part!).
Your errors seem to happen around the same area (~20K docs). If you
skip the first say ~18K docs does the error still happen? We need to
somehow narrow this down.
Or is there any way I could get a temporary account to log into this
box and try to track this down? (If indeed it doesn't happen on an
x86 box -- I unfortunately don't have access to a PPC machine).
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
So, it's a little clearer. I get the Array-out-of-bounds exception if
I'm re-indexing some already indexed documents -- if there are
deletions involved. I get the CorruptIndexException if I'm indexing
freshly -- no deletions. Here's an example of that (with the latest
nightly). I removed the existing index, then reindexed the collection
six UpLib docs at a time, till I hit the corruption.
Bill
/Library/Java/Home/bin/java -Dcom.parc.uplib.indexing.debugMode=true "-Dcom.parc.uplib.indexing.indexProperties=contents:title:categories$,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-message-id*:email-guid*:email-subject:email-from-name:email-from-address*:email-attachment-to*:email-thread-index*:email-references$,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*" -classpath "/local/uplib/share/UpLib-1.7/code/lucene-core-2.3-2007-11-29_02-49-31.jar:/local/uplib/share/UpLib-1.7/code/LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000 com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index" update /local/janssen-uplib/docs 01113-86-6099-767 01113-86-5485-936 01113-86-0975-795 01113-62-2881-882 01113-44-7730-580 01113-44-7684-477
thr002: acquiring lock: LuceneIndex...
thr002: acquired lock: LuceneIndex*
thr002: releasing lock: LuceneIndex*
thr002: indexing output is <updating
doc_root_dir is /local/janssen-uplib/docs
index file is /local/janssen-uplib/index and it exists.
Deleted 0 existing instances of 01113-86-6099-767
Deleted 0 existing instances of 01113-86-5485-936
Deleted 0 existing instances of 01113-86-0975-795
Deleted 0 existing instances of 01113-62-2881-882
Deleted 0 existing instances of 01113-44-7730-580
Deleted 0 existing instances of 01113-44-7684-477
IFD [main]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4865ce
IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/local/janssen-uplib/index autoCommit=true mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@4ae2c1 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6d2702 ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=_94:c8686
IW 0 [main]: setMaxFieldLength 2147483647
Working on document /local/janssen-uplib/docs/01113-86-6099-767
Adding header 'abstract' IT to 01113-86-6099-767
Adding header 'apparent-mime-type' I to 01113-86-6099-767
Adding header 'authors' IT to 01113-86-6099-767
Adding header 'categories' I (flowport) to 01113-86-6099-767
Adding header 'categories' I (article) to 01113-86-6099-767
Adding header 'categories' I (flowport) to 01113-86-6099-767
Adding header 'categories' I (flowport) to 01113-86-6099-767
Adding header 'citation' I to 01113-86-6099-767
Adding header 'date' I (20030800) to 01113-86-6099-767
Adding header 'sha-hash' I to 01113-86-6099-767
Adding header 'title' IT (The Great Chive Division) to 01113-86-6099-767
Created empty doc Document<stored/uncompressed,indexed<id:01113-86-6099-767> stored/uncompressed,indexed<uplibdate:20050418> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (4219): Question: My chives have grown
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-86-6099-767 (2 versions)
Working on document /local/janssen-uplib/docs/01113-86-5485-936
Adding header 'abstract' IT to 01113-86-5485-936
Adding header 'apparent-mime-type' I to 01113-86-5485-936
Adding header 'authors' IT to 01113-86-5485-936
Adding header 'categories' I (paper) to 01113-86-5485-936
Adding header 'categories' I (sensepad) to 01113-86-5485-936
Adding header 'citation' I to 01113-86-5485-936
Adding header 'date' I (20040524) to 01113-86-5485-936
Adding header 'sha-hash' I to 01113-86-5485-936
Adding header 'title' IT (Designing Interaction, not Interfaces) to 01113-86-5485-936
Created empty doc Document<stored/uncompressed,indexed<id:01113-86-5485-936> stored/uncompressed,indexed<uplibdate:20050418> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3855): Designing Interaction, not Int
page 1 (5688): Figure 1. Interaction as a phe
page 2 (5831): Interaction models can be eval
page 3 (5770): Reification turns concepts and
page 4 (5558): Figure 6. A mock-up of the DPI
page 5 (5963): In joint work with Yves Guiard
page 6 (6819): I propose making interactions
page 7 (5622): Graphical Application. Proc. A
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-86-5485-936 (9 versions)
Working on document /local/janssen-uplib/docs/01113-86-0975-795
Adding header 'apparent-mime-type' I to 01113-86-0975-795
Adding header 'categories' I (article) to 01113-86-0975-795
Adding header 'date' I (20050414) to 01113-86-0975-795
Adding header 'sha-hash' I to 01113-86-0975-795
Adding header 'source' IT to 01113-86-0975-795
Created empty doc Document<stored/uncompressed,indexed<id:01113-86-0975-795> stored/uncompressed,indexed<uplibdate:20050418> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (1851): About sponsorship Simplifying
page 1 (2900): Latvia and Lithuania, Estonia'
page 2 (3088): How much fairness is gained fo
page 3 (5317): At the time of its reform, Est
page 4 (1101): In part, the tax system is bur
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-86-0975-795 (6 versions)
Working on document /local/janssen-uplib/docs/01113-62-2881-882
Adding header 'apparent-mime-type' I to 01113-62-2881-882
Adding header 'categories' I (article) to 01113-62-2881-882
Adding header 'date' I (20050328) to 01113-62-2881-882
Adding header 'keywords' I (neuroeconomics) to 01113-62-2881-882
Adding header 'sha-hash' I to 01113-62-2881-882
Adding header 'title' IT (Neuroeconomics: Why Logic Often Takes a Backseat) to 01113-62-2881-882
Created empty doc Document<stored/uncompressed,indexed<id:01113-62-2881-882> stored/uncompressed,indexed<uplibdate:20050415> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (2957): Close Window MARCH 28, 2005 EC
page 1 (3856): these attacks on rationality ?
page 2 (484): Even believers in neuroeconomi
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-62-2881-882 (4 versions)
Working on document /local/janssen-uplib/docs/01113-44-7730-580
Adding header 'apparent-mime-type' I to 01113-44-7730-580
Adding header 'categories' I (flowport) to 01113-44-7730-580
Adding header 'categories' I (receipt) to 01113-44-7730-580
Adding header 'categories' I (flowport) to 01113-44-7730-580
Adding header 'categories' I (flowport) to 01113-44-7730-580
Adding header 'comment' IT to 01113-44-7730-580
Adding header 'sha-hash' I to 01113-44-7730-580
Adding header 'title' IT (fax receipt for JCDL 2005 demo submission document) to 01113-44-7730-580
Created empty doc Document<stored/uncompressed,indexed<id:01113-44-7730-580> stored/uncompressed,indexed<uplibdate:20050413> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3383): Transmission fleport Dare T lo
page 1 (2697): ACM Permission and Release For
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-44-7730-580 (3 versions)
Working on document /local/janssen-uplib/docs/01113-44-7684-477
Adding header 'apparent-mime-type' I to 01113-44-7684-477
Adding header 'authors' IT to 01113-44-7684-477
Adding header 'categories' I (flowport) to 01113-44-7684-477
Adding header 'categories' I (article) to 01113-44-7684-477
Adding header 'categories' I (ebook) to 01113-44-7684-477
Adding header 'categories' I (flowport) to 01113-44-7684-477
Adding header 'categories' I (flowport) to 01113-44-7684-477
Adding header 'citation' I to 01113-44-7684-477
Adding header 'date' I (20050117) to 01113-44-7684-477
Adding header 'sha-hash' I to 01113-44-7684-477
Adding header 'title' IT (Signs of Life for E-books in 2004) to 01113-44-7684-477
Created empty doc Document<stored/uncompressed,indexed<id:01113-44-7684-477> stored/uncompressed,indexed<uplibdate:20050413> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (6514): Signs ofLife for E-books in 20
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01113-44-7684-477 (2 versions)
Optimizing...
IW 0 [main]: optimize: index now _94:c8686
IW 0 [main]: flush: segment=_95 docStoreSegment=_95 docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=true numDocs=26 numBufDelTerms=0
IW 0 [main]: index before flush _94:c8686
closeDocStore: 2 files to flush to segment _95
flush postings as segment _95 numDocs=26
oldRAMSize=269104 newFlushedSize=114988 docs/MB=237.094 new/old=42.73%
IW 0 [main]: checkpoint: wrote segments file "segments_ic"
IFD [main]: now checkpoint "segments_ic" [2 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_ib"
IFD [main]: delete "segments_ib"
IW 0 [main]: checkpoint: wrote segments file "segments_id"
IFD [main]: now checkpoint "segments_id" [2 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_ic"
IFD [main]: delete "_95.fnm"
IFD [main]: delete "_95.frq"
IFD [main]: delete "_95.prx"
IFD [main]: delete "_95.tis"
IFD [main]: delete "_95.tii"
IFD [main]: delete "_95.nrm"
IFD [main]: delete "_95.fdx"
IFD [main]: delete "_95.fdt"
IFD [main]: delete "segments_ic"
IW 0 [main]: LMP: findMerges: 2 segments
IW 0 [main]: LMP: level 6.4501038 to 7.2001038: 1 segments
IW 0 [main]: LMP: level -1.0 to 5.070234: 1 segments
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _94:c8686 _95:c26
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [main]: add merge to pendingMerges: _94:c8686 _95:c26 [optimize] [total 1 pending]
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _94:c8686 _95:c26
IW 0 [main]: CMS: consider merge _94:c8686 _95:c26 into _96 [optimize]
IW 0 [main]: CMS: launch new thread [Thread-0]
IW 0 [Thread-0]: CMS: merge thread: start
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [Thread-0]: now merge
merge=_94:c8686 _95:c26 into _96 [optimize]
index=_94:c8686 _95:c26
IW 0 [Thread-0]: merging _94:c8686 _95:c26 into _96 [optimize]
IW 0 [Thread-0]: merge: total 8712 docs
IW 0 [Thread-0]: hit exception during merge; now refresh deleter on segment _96
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.fdt"
IFD [Thread-0]: delete "_96.fdt"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.fdx"
IFD [Thread-0]: delete "_96.fdx"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.fnm"
IFD [Thread-0]: delete "_96.fnm"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.frq"
IFD [Thread-0]: delete "_96.frq"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.prx"
IFD [Thread-0]: delete "_96.prx"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.tii"
IFD [Thread-0]: delete "_96.tii"
IFD [Thread-0]: refresh [prefix=_96]: removing newly created unreferenced file "_96.tis"
IFD [Thread-0]: delete "_96.tis"
IW 0 [Thread-0]: hit exception during merge
java.io.IOException: background merge hit exception: _94:c8686 _95:c26 into _96 [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1705)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1654)
at com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:414)
at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:659)
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order (8692 <= 10221 )
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:474)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
Exception in thread "Thread-0" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: docs out of order (8692 <= 10221 )
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:274)
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order (8692 <= 10221 )
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:474)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is in the nightly JAR. It's o.a.l.index.CheckIndex (it defines
a static main).
Mike
"Bill Janssen" <ja...@parc.com> wrote:
> > Also, could you try out the CheckIndex tool in 2.3-dev before and
> > after the deletes?
>
> Great idea! I don't suppose there's a jar file of it?
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Also, could you try out the CheckIndex tool in 2.3-dev before and
> after the deletes?
Great idea! I don't suppose there's a jar file of it?
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
On Nov 29, 2007, at 2:26 PM, Bill Janssen wrote:
>> Are you still getting the original exception too or just the Array
>> out =20=
>>
>> of bounds one now? Also, are you doing anything else to the index
>> =20
>> while this is happening? The code at the point in the exception
>> below =20=
>>
>> is trying to properly handle deleted documents.
>
> Just the array-out-of-bounds one, now. The current version of the
> code creates a writer, then deletes all old Lucene 'Document'
> instances belonging to the specified UpLib doc-id, using that writer,
> then re-indexes that UpLib doc-id (which consists of one-to-N Lucene
> 'Document's). After doing the six UpLib documents, it calls
> optimize().
I'm curious what happens if you call optimize after doing the deletion
but before the re-indexing.
Also, could you try out the CheckIndex tool in 2.3-dev before and
after the deletes?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Could you post this part of the code (deleting) too?
Here it is:
private static void remove (File index_file, String[] doc_ids, int start) {
String number;
String list;
Term term;
TermDocs matches;
if (debug_mode)
System.err.println("index file is " + index_file + " and it " + (index_file.exists() ? "exists." : "does not exist."));
try {
if (index_file.exists() && (doc_ids.length > start)) {
IndexReader reader = IndexReader.open(index_file);
try {
for (int i = start; i < doc_ids.length; i++) {
term = new Term("id", doc_ids[i]);
int deleted = reader.deleteDocuments(term);
System.out.println("Deleted " + deleted + " existing instances of " + doc_ids[i]);
}
} finally {
reader.close();
}
}
} catch (Exception e) {
if (debug_mode) {
e.printStackTrace(System.err);
} else {
System.out.println("* LuceneIndexing 'remove' raised " + e.getClass() + " with message " + e.getMessage());
System.err.println("LuceneIndexing 'remove': caught a " + e.getClass() +
"\n with message: " + e.getMessage());
System.out.flush();
}
System.exit(JAVA_EXCEPTION);
}
System.out.flush();
}
private static void update (File index_file, File doc_root_dir, String[] ids, int start) {
ExtractIndexingInfo.DocumentIterator docit;
String number;
remove (index_file, ids, start);
try {
// Now add the documents to the index
IndexWriter writer = new IndexWriter(index_file, new StandardAnalyzer(), !index_file.exists());
if (debug_mode)
writer.setInfoStream(System.err);
writer.setMaxFieldLength(Integer.MAX_VALUE);
try {
for (int i = start; i < ids.length; i ++) {
docit = build_document_iterator(doc_root_dir, ids[i]);
int count = 0;
while (docit.hasNext()) {
writer.addDocument((Document)(docit.next()));
count += 1;
}
System.out.println("Added " + docit.id + " (" + count + " versions)");
System.out.flush();
}
} finally {
// And close the index
System.out.println("Optimizing...");
// See http://www.gossamer-threads.com/lists/lucene/java-dev/47895 about optimize
// Can fail if low on disk space
writer.optimize();
writer.close();
}
} catch (Exception e) {
if (debug_mode) {
e.printStackTrace(System.err);
} else {
System.out.println("* Lucene search engine raised " + e.getClass() + " with message " + e.getMessage());
System.err.println(" 'update' caught a " + e.getClass() +
"\n with message: " + e.getMessage());
System.out.flush();
}
System.exit(JAVA_EXCEPTION);
}
System.out.flush();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Grant Ingersoll" <gs...@apache.org> wrote:
> Just a theory (make that a guess), Mike, but is it possible that the
> one merge scheduler is hitting a synchronization issue with the
> deletedDocuments bit vector? That is one thread is cleaning it up and
> the other is accessing and they aren't synchronizing their access?
Well, in trunk I think we are hitting the bit vector in synchronized
contexts, correctly. (I sure think/hope so :). Also, in the context
of merging, the deleted docs bit vector is read only.
This sure does spookily sound like LUCENE-140!! I hope that one is not
coming back from the dead!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
Just a theory (make that a guess), Mike, but is it possible that the
one merge scheduler is hitting a synchronization issue with the
deletedDocuments bit vector? That is one thread is cleaning it up and
the other is accessing and they aren't synchronizing their access?
This doesn't explain the original problem, but maybe this one?
On Nov 29, 2007, at 4:46 PM, Bill Janssen wrote:
>> Have you tried another PPC machine?
>
> No. It's in another location, but perhaps I can get it tomorrow. On
> the other hand, the success when using 2.0 makes it likely to me that
> the machine isn't the problem.
>
> OK, I've reverted to my original codebase (where I first create a
> reader and do the deletions, then create a writer and do the additions
> and optimize), and it works fine with lucene-core-2.0.0, but fails
> with lucene-core-2.3.-whatever (last night's build). Here's the dump:
>
> indexing with /Library/Java/Home/bin/java -
> Dcom.parc.uplib.indexing.debugMode=true "-
> Dcom.parc.uplib.indexing.indexProperties=contents:title:categories
> $,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-
> message-id*:email-guid*:email-subject:email-from-name:email-from-
> address*:email-attachment-to*:email-thread-index*:email-references
> $,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-
> genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*" -
> classpath "/local/uplib/share/UpLib-1.7/code/lucene-
> core-2.3-2007-11-29_02-49-31.jar:/local/uplib/share/UpLib-1.7/code/
> LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000
> com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index"
> update /local/janssen-uplib/docs 01160-06-3246-773 01159-97-2914-663
> 01159-89-7507-719 01159-89-5614-073 01159-89-1159-244
> 01159-89-0665-499
> thr001: acquiring lock: LuceneIndex...
> thr001: acquired lock: LuceneIndex*
> thr001: releasing lock: LuceneIndex*
> thr001: indexing output is <updating
> doc_root_dir is /local/janssen-uplib/docs
> index file is /local/janssen-uplib/index and it exists.
> Deleted 1 existing instances of 01160-06-3246-773
> Deleted 1 existing instances of 01159-97-2914-663
> Deleted 5 existing instances of 01159-89-7507-719
> Deleted 26 existing instances of 01159-89-5614-073
> Deleted 5 existing instances of 01159-89-1159-244
> Deleted 6 existing instances of 01159-89-0665-499
> IFD [main]: setInfoStream
> deletionPolicy
> =org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@9b42e6
> IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/
> local/janssen-uplib/index autoCommit=true
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@6c79d7
> mergeScheduler
> =org.apache.lucene.index.ConcurrentMergeScheduler@b33d0a
> ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=_4j:c19686
> IW 0 [main]: setMaxFieldLength 2147483647
> Working on document /local/janssen-uplib/docs/01160-06-3246-773
> Adding header 'apparent-mime-type' I to 01160-06-3246-773
> Adding header 'authors' IT to 01160-06-3246-773
> Adding header 'categories' I (cartoon) to 01160-06-3246-773
> Adding header 'date' I (19951005) to 01160-06-3246-773
> Adding header 'sha-hash' I to 01160-06-3246-773
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01160-06-3246-773> stored/uncompressed,indexed<uplibdate:20061005>
> stored/uncompressed,indexed<uplibtype:whole>>
> Added 01160-06-3246-773 (1 versions)
> Working on document /local/janssen-uplib/docs/01159-97-2914-663
> Adding header 'apparent-mime-type' I to 01159-97-2914-663
> Adding header 'authors' IT to 01159-97-2914-663
> Adding header 'categories' I (cartoon) to 01159-97-2914-663
> Adding header 'date' I (19951004) to 01159-97-2914-663
> Adding header 'sha-hash' I to 01159-97-2914-663
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01159-97-2914-663> stored/uncompressed,indexed<uplibdate:20061004>
> stored/uncompressed,indexed<uplibtype:whole>>
> Added 01159-97-2914-663 (1 versions)
> Working on document /local/janssen-uplib/docs/01159-89-7507-719
> Adding header 'apparent-mime-type' I to 01159-89-7507-719
> Adding header 'sha-hash' I to 01159-89-7507-719
> Adding header 'title' IT (Photoshop Metal Texture) to
> 01159-89-7507-719
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01159-89-7507-719> stored/uncompressed,indexed<uplibdate:20061003>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (580): Tutorials\xa5 News\xa5 Exclusives\xa5 S
> page 1 (1680): On a new layer create a gradie
> page 2 (1118): Scrapes and scratches are irre
> page 3 (470): Bevel Settings\xa5 Contour Settin
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01159-89-7507-719 (5 versions)
> Working on document /local/janssen-uplib/docs/01159-89-5614-073
> Adding header 'apparent-mime-type' I to 01159-89-5614-073
> Adding header 'sha-hash' I to 01159-89-5614-073
> Adding header 'title' IT (Creating Virtual Mats and Frames with The
> GIMP) to 01159-89-5614-073
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01159-89-5614-073> stored/uncompressed,indexed<uplibdate:20061003>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (663): All photographs and articles o
> page 1 (600): Although real mats and frames
> page 2 (999): The Procedure First of all you
> page 3 (615): Run the Add Mat script (Script
> page 4 (693): in the GIMP toolbox Pattern: U
> page 5 (703): 3D lighted/shaded appearance.
> page 6 (719): Bevel Fill Color, pops up a di
> page 7 (714): texture afterwards. Default: o
> page 8 (797): recommended, especially if you
> page 9 (461): moving outwards, as in adding
> page 10 (67): 11 Creating Virtual Mats and F
> page 11 (67): 12 Creating Virtual Mats and F
> page 12 (378): Time to add a frame. Run Scrip
> page 13 (498): in Frame Fill Color FG color:
> page 14 (717): and background colors, not in
> page 15 (685): the pattern to for texturing t
> page 16 (721): added along the inner boundary
> page 17 (1006): leave a selection in place cov
> page 18 (904): A drop shadow on the entire fr
> page 19 (629): threshold sliders to the right
> page 20 (901): Bump Map" and fill it with whi
> page 21 (786): image window, do a Select All
> page 22 (393): In the Layers dialog, choose t
> page 23 (937): "Keep Trans." option near the
> page 24 (239): Last modified: Mon May 9 23:36
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01159-89-5614-073 (26 versions)
> Working on document /local/janssen-uplib/docs/01159-89-1159-244
> Adding header 'apparent-mime-type' I to 01159-89-1159-244
> Adding header 'authors' IT to 01159-89-1159-244
> Adding header 'categories' I (ebooks) to 01159-89-1159-244
> Adding header 'categories' I (article) to 01159-89-1159-244
> Adding header 'date' I (20050100) to 01159-89-1159-244
> Adding header 'sha-hash' I to 01159-89-1159-244
> Adding header 'title' IT (The Future of Books) to 01159-89-1159-244
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01159-89-1159-244> stored/uncompressed,indexed<uplibdate:20061003>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (3649): Close Window The Future of Boo
> page 1 (4291): Ken agreed and suggested that
> page 2 (3934): Catalog online but instead auc
> page 3 (2331): At Marsh's workshop we watched
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01159-89-1159-244 (5 versions)
> Working on document /local/janssen-uplib/docs/01159-89-0665-499
> Adding header 'apparent-mime-type' I to 01159-89-0665-499
> Adding header 'authors' IT to 01159-89-0665-499
> Adding header 'categories' I (review) to 01159-89-0665-499
> Adding header 'categories' I (article) to 01159-89-0665-499
> Adding header 'categories' I (ebooks) to 01159-89-0665-499
> Adding header 'date' I (20061019) to 01159-89-0665-499
> Adding header 'sha-hash' I to 01159-89-0665-499
> Adding header 'title' IT (Books@Google) to 01159-89-0665-499
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01159-89-0665-499> stored/uncompressed,indexed<uplibdate:20061003>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (1595): Home \xe1 Your account \xe1 Current
> page 1 (4636): sapienscompelled the invention
> page 2 (4341): to a base population of 40,000
> page 3 (5113): The privacy policy and the Chi
> page 4 (792): Notes [1] Wikipedia, unlike Go
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01159-89-0665-499 (6 versions)
> Optimizing...
> IW 0 [main]: optimize: index now _4j:c19686
> IW 0 [main]: flush: segment=_4k docStoreSegment=_4k
> docStoreOffset=0 flushDocs=true flushDeletes=false
> flushDocStores=true numDocs=44 numBufDelTerms=0
> IW 0 [main]: index before flush _4j:c19686
>
> closeDocStore: 2 files to flush to segment _4k
>
> flush postings as segment _4k numDocs=44
> oldRAMSize=141248 newFlushedSize=67003 docs/MB=688.586 new/
> old=47.436%
> IW 0 [main]: checkpoint: wrote segments file "segments_be"
> IFD [main]: now checkpoint "segments_be" [2 segments ; isCommit =
> true]
> IFD [main]: deleteCommits: now remove commit "segments_bd"
> IFD [main]: delete "segments_bd"
> IW 0 [main]: checkpoint: wrote segments file "segments_bf"
> IFD [main]: now checkpoint "segments_bf" [2 segments ; isCommit =
> true]
> IFD [main]: deleteCommits: now remove commit "segments_be"
> IFD [main]: delete "_4k.fnm"
> IFD [main]: delete "_4k.frq"
> IFD [main]: delete "_4k.prx"
> IFD [main]: delete "_4k.tis"
> IFD [main]: delete "_4k.tii"
> IFD [main]: delete "_4k.nrm"
> IFD [main]: delete "_4k.fdx"
> IFD [main]: delete "_4k.fdt"
> IFD [main]: delete "segments_be"
> IW 0 [main]: LMP: findMerges: 2 segments
> IW 0 [main]: LMP: level 6.744767 to 7.494767: 1 segments
> IW 0 [main]: LMP: level -1.0 to 4.842865: 1 segments
> IW 0 [main]: CMS: now merge
> IW 0 [main]: CMS: index: _4j:c19686 _4k:c44
> IW 0 [main]: CMS: no more merges pending; now return
> IW 0 [main]: add merge to pendingMerges: _4j:c19686 _4k:c44
> [optimize] [total 1 pending]
> IW 0 [main]: CMS: now merge
> IW 0 [main]: CMS: index: _4j:c19686 _4k:c44
> IW 0 [main]: CMS: consider merge _4j:c19686 _4k:c44 into _4l
> [optimize]
> IW 0 [main]: CMS: launch new thread [Thread-0]
> IW 0 [main]: CMS: no more merges pending; now return
> IW 0 [Thread-0]: CMS: merge thread: start
> IW 0 [Thread-0]: now merge
> merge=_4j:c19686 _4k:c44 into _4l [optimize]
> index=_4j:c19686 _4k:c44
> IW 0 [Thread-0]: merging _4j:c19686 _4k:c44 into _4l [optimize]
> IW 0 [Thread-0]: merge: total 19686 docs
> IW 0 [Thread-0]: hit exception during merge; now refresh deleter on
> segment _4l
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.fdt"
> IFD [Thread-0]: delete "_4l.fdt"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.fdx"
> IFD [Thread-0]: delete "_4l.fdx"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.fnm"
> IFD [Thread-0]: delete "_4l.fnm"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.frq"
> IFD [Thread-0]: delete "_4l.frq"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.prx"
> IFD [Thread-0]: delete "_4l.prx"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.tii"
> IFD [Thread-0]: delete "_4l.tii"
> IFD [Thread-0]: refresh [prefix=_4l]: removing newly created
> unreferenced file "_4l.tis"
> IFD [Thread-0]: delete "_4l.tis"
> IW 0 [Thread-0]: hit exception during merge
> java.io.IOException: background merge hit exception: _4j:c19686
> _4k:c44 into _4l [optimize]
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
> 1705)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
> 1654)
> at
> com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:414)
> at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:
> 659)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out
> of range: 21352
> at org.apache.lucene.util.BitVector.get(BitVector.java:72)
> at
> org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
> at
> org
> .apache
> .lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
> at
> org
> .apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:
> 467)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:
> 430)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:
> 402)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:
> 366)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
> 123)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:
> 3002)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:240)
> Exception in thread "Thread-0" org.apache.lucene.index.MergePolicy
> $MergeException: java.lang.ArrayIndexOutOfBoundsException: Array
> index out of range: 21352
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:274)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out
> of range: 21352
> at org.apache.lucene.util.BitVector.get(BitVector.java:72)
> at
> org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
> at
> org
> .apache
> .lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
> at
> org
> .apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:
> 467)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:
> 430)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:
> 402)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:
> 366)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
> 123)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:
> 3002)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:240)
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Have you tried another PPC machine?
No. It's in another location, but perhaps I can get it tomorrow. On
the other hand, the success when using 2.0 makes it likely to me that
the machine isn't the problem.
OK, I've reverted to my original codebase (where I first create a
reader and do the deletions, then create a writer and do the additions
and optimize), and it works fine with lucene-core-2.0.0, but fails
with lucene-core-2.3.-whatever (last night's build). Here's the dump:
indexing with /Library/Java/Home/bin/java -Dcom.parc.uplib.indexing.debugMode=true "-Dcom.parc.uplib.indexing.indexProperties=contents:title:categories$,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-message-id*:email-guid*:email-subject:email-from-name:email-from-address*:email-attachment-to*:email-thread-index*:email-references$,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*" -classpath "/local/uplib/share/UpLib-1.7/code/lucene-core-2.3-2007-11-29_02-49-31.jar:/local/uplib/share/UpLib-1.7/code/LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000 com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index" update /local/janssen-uplib/docs 01160-06-3246-773 01159-97-2914-663 01159-89-7507-719 01159-89-5614-073 01159-89-1159-244 01159-89-0665-499
thr001: acquiring lock: LuceneIndex...
thr001: acquired lock: LuceneIndex*
thr001: releasing lock: LuceneIndex*
thr001: indexing output is <updating
doc_root_dir is /local/janssen-uplib/docs
index file is /local/janssen-uplib/index and it exists.
Deleted 1 existing instances of 01160-06-3246-773
Deleted 1 existing instances of 01159-97-2914-663
Deleted 5 existing instances of 01159-89-7507-719
Deleted 26 existing instances of 01159-89-5614-073
Deleted 5 existing instances of 01159-89-1159-244
Deleted 6 existing instances of 01159-89-0665-499
IFD [main]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@9b42e6
IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/local/janssen-uplib/index autoCommit=true mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@6c79d7 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@b33d0a ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=_4j:c19686
IW 0 [main]: setMaxFieldLength 2147483647
Working on document /local/janssen-uplib/docs/01160-06-3246-773
Adding header 'apparent-mime-type' I to 01160-06-3246-773
Adding header 'authors' IT to 01160-06-3246-773
Adding header 'categories' I (cartoon) to 01160-06-3246-773
Adding header 'date' I (19951005) to 01160-06-3246-773
Adding header 'sha-hash' I to 01160-06-3246-773
Created empty doc Document<stored/uncompressed,indexed<id:01160-06-3246-773> stored/uncompressed,indexed<uplibdate:20061005> stored/uncompressed,indexed<uplibtype:whole>>
Added 01160-06-3246-773 (1 versions)
Working on document /local/janssen-uplib/docs/01159-97-2914-663
Adding header 'apparent-mime-type' I to 01159-97-2914-663
Adding header 'authors' IT to 01159-97-2914-663
Adding header 'categories' I (cartoon) to 01159-97-2914-663
Adding header 'date' I (19951004) to 01159-97-2914-663
Adding header 'sha-hash' I to 01159-97-2914-663
Created empty doc Document<stored/uncompressed,indexed<id:01159-97-2914-663> stored/uncompressed,indexed<uplibdate:20061004> stored/uncompressed,indexed<uplibtype:whole>>
Added 01159-97-2914-663 (1 versions)
Working on document /local/janssen-uplib/docs/01159-89-7507-719
Adding header 'apparent-mime-type' I to 01159-89-7507-719
Adding header 'sha-hash' I to 01159-89-7507-719
Adding header 'title' IT (Photoshop Metal Texture) to 01159-89-7507-719
Created empty doc Document<stored/uncompressed,indexed<id:01159-89-7507-719> stored/uncompressed,indexed<uplibdate:20061003> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (580): Tutorials\xa5 News\xa5 Exclusives\xa5 S
page 1 (1680): On a new layer create a gradie
page 2 (1118): Scrapes and scratches are irre
page 3 (470): Bevel Settings\xa5 Contour Settin
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01159-89-7507-719 (5 versions)
Working on document /local/janssen-uplib/docs/01159-89-5614-073
Adding header 'apparent-mime-type' I to 01159-89-5614-073
Adding header 'sha-hash' I to 01159-89-5614-073
Adding header 'title' IT (Creating Virtual Mats and Frames with The GIMP) to 01159-89-5614-073
Created empty doc Document<stored/uncompressed,indexed<id:01159-89-5614-073> stored/uncompressed,indexed<uplibdate:20061003> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (663): All photographs and articles o
page 1 (600): Although real mats and frames
page 2 (999): The Procedure First of all you
page 3 (615): Run the Add Mat script (Script
page 4 (693): in the GIMP toolbox Pattern: U
page 5 (703): 3D lighted/shaded appearance.
page 6 (719): Bevel Fill Color, pops up a di
page 7 (714): texture afterwards. Default: o
page 8 (797): recommended, especially if you
page 9 (461): moving outwards, as in adding
page 10 (67): 11 Creating Virtual Mats and F
page 11 (67): 12 Creating Virtual Mats and F
page 12 (378): Time to add a frame. Run Scrip
page 13 (498): in Frame Fill Color FG color:
page 14 (717): and background colors, not in
page 15 (685): the pattern to for texturing t
page 16 (721): added along the inner boundary
page 17 (1006): leave a selection in place cov
page 18 (904): A drop shadow on the entire fr
page 19 (629): threshold sliders to the right
page 20 (901): Bump Map" and fill it with whi
page 21 (786): image window, do a Select All
page 22 (393): In the Layers dialog, choose t
page 23 (937): "Keep Trans." option near the
page 24 (239): Last modified: Mon May 9 23:36
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01159-89-5614-073 (26 versions)
Working on document /local/janssen-uplib/docs/01159-89-1159-244
Adding header 'apparent-mime-type' I to 01159-89-1159-244
Adding header 'authors' IT to 01159-89-1159-244
Adding header 'categories' I (ebooks) to 01159-89-1159-244
Adding header 'categories' I (article) to 01159-89-1159-244
Adding header 'date' I (20050100) to 01159-89-1159-244
Adding header 'sha-hash' I to 01159-89-1159-244
Adding header 'title' IT (The Future of Books) to 01159-89-1159-244
Created empty doc Document<stored/uncompressed,indexed<id:01159-89-1159-244> stored/uncompressed,indexed<uplibdate:20061003> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3649): Close Window The Future of Boo
page 1 (4291): Ken agreed and suggested that
page 2 (3934): Catalog online but instead auc
page 3 (2331): At Marsh's workshop we watched
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01159-89-1159-244 (5 versions)
Working on document /local/janssen-uplib/docs/01159-89-0665-499
Adding header 'apparent-mime-type' I to 01159-89-0665-499
Adding header 'authors' IT to 01159-89-0665-499
Adding header 'categories' I (review) to 01159-89-0665-499
Adding header 'categories' I (article) to 01159-89-0665-499
Adding header 'categories' I (ebooks) to 01159-89-0665-499
Adding header 'date' I (20061019) to 01159-89-0665-499
Adding header 'sha-hash' I to 01159-89-0665-499
Adding header 'title' IT (Books@Google) to 01159-89-0665-499
Created empty doc Document<stored/uncompressed,indexed<id:01159-89-0665-499> stored/uncompressed,indexed<uplibdate:20061003> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (1595): Home \xe1 Your account \xe1 Current
page 1 (4636): sapienscompelled the invention
page 2 (4341): to a base population of 40,000
page 3 (5113): The privacy policy and the Chi
page 4 (792): Notes [1] Wikipedia, unlike Go
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01159-89-0665-499 (6 versions)
Optimizing...
IW 0 [main]: optimize: index now _4j:c19686
IW 0 [main]: flush: segment=_4k docStoreSegment=_4k docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=true numDocs=44 numBufDelTerms=0
IW 0 [main]: index before flush _4j:c19686
closeDocStore: 2 files to flush to segment _4k
flush postings as segment _4k numDocs=44
oldRAMSize=141248 newFlushedSize=67003 docs/MB=688.586 new/old=47.436%
IW 0 [main]: checkpoint: wrote segments file "segments_be"
IFD [main]: now checkpoint "segments_be" [2 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_bd"
IFD [main]: delete "segments_bd"
IW 0 [main]: checkpoint: wrote segments file "segments_bf"
IFD [main]: now checkpoint "segments_bf" [2 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_be"
IFD [main]: delete "_4k.fnm"
IFD [main]: delete "_4k.frq"
IFD [main]: delete "_4k.prx"
IFD [main]: delete "_4k.tis"
IFD [main]: delete "_4k.tii"
IFD [main]: delete "_4k.nrm"
IFD [main]: delete "_4k.fdx"
IFD [main]: delete "_4k.fdt"
IFD [main]: delete "segments_be"
IW 0 [main]: LMP: findMerges: 2 segments
IW 0 [main]: LMP: level 6.744767 to 7.494767: 1 segments
IW 0 [main]: LMP: level -1.0 to 4.842865: 1 segments
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _4j:c19686 _4k:c44
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [main]: add merge to pendingMerges: _4j:c19686 _4k:c44 [optimize] [total 1 pending]
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _4j:c19686 _4k:c44
IW 0 [main]: CMS: consider merge _4j:c19686 _4k:c44 into _4l [optimize]
IW 0 [main]: CMS: launch new thread [Thread-0]
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [Thread-0]: CMS: merge thread: start
IW 0 [Thread-0]: now merge
merge=_4j:c19686 _4k:c44 into _4l [optimize]
index=_4j:c19686 _4k:c44
IW 0 [Thread-0]: merging _4j:c19686 _4k:c44 into _4l [optimize]
IW 0 [Thread-0]: merge: total 19686 docs
IW 0 [Thread-0]: hit exception during merge; now refresh deleter on segment _4l
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.fdt"
IFD [Thread-0]: delete "_4l.fdt"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.fdx"
IFD [Thread-0]: delete "_4l.fdx"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.fnm"
IFD [Thread-0]: delete "_4l.fnm"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.frq"
IFD [Thread-0]: delete "_4l.frq"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.prx"
IFD [Thread-0]: delete "_4l.prx"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.tii"
IFD [Thread-0]: delete "_4l.tii"
IFD [Thread-0]: refresh [prefix=_4l]: removing newly created unreferenced file "_4l.tis"
IFD [Thread-0]: delete "_4l.tis"
IW 0 [Thread-0]: hit exception during merge
java.io.IOException: background merge hit exception: _4j:c19686 _4k:c44 into _4l [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1705)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1654)
at com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:414)
at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:659)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 21352
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:467)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
Exception in thread "Thread-0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 21352
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:274)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 21352
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:467)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Bill Janssen" <ja...@parc.com> wrote:
> Here's the dump with last night's build:
Those logs look healthy up until the exception.
One odd thing is when you instantiate your writer, your index has 2
segments in it. I expected only 1 since each time you visit your
index you leave it optimized. (Or, maybe you're not calling
setInfoStream immediately after opening the writer?).
The error still only happens on the one PPC machine, even after
upgrading to trunk? EG not on an Intel box?
Have you tried another PPC machine?
> I'm going back to the old code now. It uses the 2.0 APIs, so it
> uses an IndexReader to delete the existing instances, then closes
> that reader (which if I understand it properly should flush the
> index back to disk), then creates a new writer to re-index the same
> documents, then does the optimize with that writer, which is where
> the CorruptIndexException started coming up. I'm going to run that
> again with 2.0, then with last night's build.
Could you post this part of the code (deleting) too?
> I'm not sure if the success with 2.0 meant that a corrupted index
> wasn't being detected, or if it wasn't being corrupted in the first
> place.
Likely the corruption really isn't happening. That particular check
for "docs out of order" is present in 2.0 as well.
Is it possible to whittle down your test to a smaller set of documents?
EG if you only re-index one document at a time, does the exception
happen sooner? Ideally we can reduce this to a test I can reproduce
then I can track it down...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Are you still getting the original exception too or just the Array out =20=
>
> of bounds one now? Also, are you doing anything else to the index =20
> while this is happening? The code at the point in the exception below =20=
>
> is trying to properly handle deleted documents.
Just the array-out-of-bounds one, now. The current version of the
code creates a writer, then deletes all old Lucene 'Document'
instances belonging to the specified UpLib doc-id, using that writer,
then re-indexes that UpLib doc-id (which consists of one-to-N Lucene
'Document's). After doing the six UpLib documents, it calls
optimize().
I'm going back to the old code now. It uses the 2.0 APIs, so it uses
an IndexReader to delete the existing instances, then closes that
reader (which if I understand it properly should flush the index back
to disk), then creates a new writer to re-index the same documents,
then does the optimize with that writer, which is where the
CorruptIndexException started coming up. I'm going to run that again
with 2.0, then with last night's build.
I'm not sure if the success with 2.0 meant that a corrupted index
wasn't being detected, or if it wasn't being corrupted in the first
place.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Grant Ingersoll <gs...@apache.org>.
Are you still getting the original exception too or just the Array out
of bounds one now? Also, are you doing anything else to the index
while this is happening? The code at the point in the exception below
is trying to properly handle deleted documents.
-Grant
On Nov 29, 2007, at 1:34 PM, Bill Janssen wrote:
>> Can you try running with the trunk version of Lucene (2.3-dev) and
>> see
>> if the error still occurs? EG you can download this AM's build here:
>>
>> http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/288/artifact/artifacts
>
> Still there. Here's the dump with last night's build:
>
> /Library/Java/Home/bin/java '-
> Dcom.parc.uplib.indexing.debugMode=true' '-
> Dcom.parc.uplib.indexing.indexProperties=contents:title:categories
> $,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-
> message-id*:email-guid*:email-subject:email-from-name:email-from-
> address*:email-attachment-to*:email-thread-index*:email-references
> $,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-
> genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*' -
> classpath "/local/uplib/share/UpLib-1.7/code/lucene-
> core-2.3-2007-11-29_02-49-31.jar:/local/uplib/share/UpLib-1.7/code/
> LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000
> com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index"
> update /local/janssen-uplib/docs 01179-00-0750-547 01178-90-9186-558
> 01178-81-4212-772 01178-81-3305-217 01178-73-1029-141
> 01178-72-8365-803
> updating
> doc_root_dir is /local/janssen-uplib/docs
> IFD [main]: setInfoStream
> deletionPolicy
> =org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@462851
> IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/
> local/janssen-uplib/index autoCommit=true
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@c56c60
> mergeScheduler
> =org.apache.lucene.index.ConcurrentMergeScheduler@4e280c
> ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=_21:c19686 _22:c92
> IW 0 [main]: setMaxFieldLength 2147483647
> Working on document /local/janssen-uplib/docs/01179-00-0750-547
> Adding header 'abstract' IT to 01179-00-0750-547
> Adding header 'apparent-mime-type' I to 01179-00-0750-547
> Adding header 'authors' IT to 01179-00-0750-547
> Adding header 'categories' I (ebooks) to 01179-00-0750-547
> Adding header 'categories' I (economics) to 01179-00-0750-547
> Adding header 'categories' I (paper) to 01179-00-0750-547
> Adding header 'citation' I to 01179-00-0750-547
> Adding header 'date' I (20070128) to 01179-00-0750-547
> Adding header 'sha-hash' I to 01179-00-0750-547
> Adding header 'title' IT (Heterogeneity in Price Stickiness and the
> Real Effects of Monetary Shocks) to 01179-00-0750-547
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01179-00-0750-547> stored/uncompressed,indexed<uplibdate:20070512>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (2181): Heterogeneity in Price Stickin
> page 1 (2927): 1 Introduction There is ample
> page 2 (3135): In the presence of strategic c
> page 3 (3128): Motivated by those questions,
> page 4 (3214): ploring the tractability of th
> page 5 (2491): model with Taylor staggered wa
> page 6 (1548): real rigidities (Ball and Rome
> page 7 (3098): 2.2 Calibrating the sectoral d
> page 8 (1913): distribution of price stickine
> page 9 (1952): reported in Table 1. Hencefort
> page 10 (1635): Figure 2 presents analogous re
> page 11 (1743): In the absence of strategic co
> page 12 (2806): Corollary 1 For an arbitrary h
> page 13 (2380): 2.4.2 Growth rate shocks In th
> page 14 (2962): price changes. With heterogene
> page 15 (3265): ties and heterogeneity in the
> page 16 (1962): complementarities. The results
> page 17 (751): to the response of the heterog
> page 18 (489): economies are embedded into th
> page 19 (3295): 2.6 Fitting IRFs with an ident
> page 20 (2066): Table 3a: Best-Fitting Duratio
> page 21 (2444): This is an important step beca
> page 22 (1976): where ? is the discount factor
> page 23 (1183): Et "? Ct+1 Ct ¦?? It Pt Pt+1 #
> page 24 (2188): can be rewritten as: Pk,t = £
> page 25 (1370): pt = Z 1 0 f (k) pk,tdk, (10)
> page 26 (3269): Heterogeneity in price stickin
> page 27 (3117): Irrespective of the net effect
> page 28 (2084): set of parameters involve high
> page 29 (575): 0 5 10 15 20 25 30 35 40 0 x 1
> page 30 (2185): output and falling prices in a
> page 31 (2358): price changes that minimizes t
> page 32 (2689): These results are fully consis
> page 33 (3600): different sources of real rigi
> page 34 (3168): work in a model with heterogen
> page 35 (2557): single equation estimation of
> page 36 (1326): Taking the limit as Æ ? 0 in e
> page 37 (1796): The output gap is constant at
> page 38 (1066): The corresponding path for the
> page 39 (1347): 4) Proof of Corollaries 1 and
> page 40 (2421): Therefore, for ? Å 0, the expe
> page 41 (1343): p (t) = Z 1 0 f (k) ? ?? ?? R
> page 42 (2117): As ? ? 0, this clearly converg
> page 43 (1375): model around the zero inflatio
> page 44 (1497): pt = Z 1 0 f (k) pk,tdk, yt =
> page 45 (1128): Table A.3: Best-Fitting Durati
> page 46 (898): Multiplying by f (k) ?k and in
> page 47 (1072): Now, from (23): ?kxk,t = pk,t
> page 48 (268): Finally, let ¹t ? pt ? pt?1 de
> page 49 (1694): References [1] Altissimo, F.,
> page 50 (1874): [14] Bils, M., P. Klenow and O
> page 51 (2091): [27] Carlton, D. (1986), ÒThe
> page 52 (1846): [39] Dixon, H. and E. Kara (20
> page 53 (1530): [51] Ohanian, L., A. Stockman
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01179-00-0750-547 (55 versions)
> Working on document /local/janssen-uplib/docs/01178-90-9186-558
> Adding header 'abstract' IT to 01178-90-9186-558
> Adding header 'apparent-mime-type' I to 01178-90-9186-558
> Adding header 'authors' IT to 01178-90-9186-558
> Adding header 'authors' IT to 01178-90-9186-558
> Adding header 'authors' IT to 01178-90-9186-558
> Adding header 'authors' IT to 01178-90-9186-558
> Adding header 'authors' IT to 01178-90-9186-558
> Adding header 'categories' I (paper) to 01178-90-9186-558
> Adding header 'categories' I (ebooks) to 01178-90-9186-558
> Adding header 'date' I (20050500) to 01178-90-9186-558
> Adding header 'sha-hash' I to 01178-90-9186-558
> Adding header 'title' IT (Visual-Syntactic Text Formatting: A New
> Method to Enhance Online) to 01178-90-9186-558
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01178-90-9186-558> stored/uncompressed,indexed<uplibdate:20070511>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (3332): Visual-Syntactic Text Formatti
> page 1 (1461): into this: To make these chang
> page 2 (4620): complex than a simple, concate
> page 3 (4723): Among some poorer readers, met
> page 4 (3827): method does not extract or dir
> page 5 (3388): (ACT scores) with gains in com
> page 6 (3693): digital text actually improve
> page 7 (3400): exam were administered. For re
> page 8 (3413): Intermediate and long-term ret
> page 9 (4219): Student preference and survey
> page 10 (4417): Discussion In print media, sim
> page 11 (4358): time; however, the opposite tr
> page 12 (3950): More time spent actually readi
> page 13 (4184): In education, the VSTF method
> page 14 (3270): References Armbruster, B.B. (2
> page 15 (3176): April 19Ð20). Neuroimaging, la
> page 16 (3350): Klare, G.R., Nichols, W.H., &
> page 17 (4098): and narrative skills connect w
> page 18 (3070): He has conducted laboratory-ba
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01178-90-9186-558 (20 versions)
> Working on document /local/janssen-uplib/docs/01178-81-4212-772
> Adding header 'abstract' IT to 01178-81-4212-772
> Adding header 'apparent-mime-type' I to 01178-81-4212-772
> Adding header 'categories' I (newspaper) to 01178-81-4212-772
> Adding header 'categories' I (article) to 01178-81-4212-772
> Adding header 'categories' I (fun) to 01178-81-4212-772
> Adding header 'categories' I (historical) to 01178-81-4212-772
> Adding header 'date' I (19340429) to 01178-81-4212-772
> Adding header 'sha-hash' I to 01178-81-4212-772
> Adding header 'title' IT (Gigantic Robots, Controlled by Wireless,
> to Fight Our Battles) to 01178-81-4212-772
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01178-81-4212-772> stored/uncompressed,indexed<uplibdate:20070510>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (261): iganlic obok ntroIIed b ireles
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01178-81-4212-772 (2 versions)
> Working on document /local/janssen-uplib/docs/01178-81-3305-217
> Adding header 'apparent-mime-type' I to 01178-81-3305-217
> Adding header 'authors' IT to 01178-81-3305-217
> Adding header 'categories' I (cartoon) to 01178-81-3305-217
> Adding header 'keywords' I (incentive) to 01178-81-3305-217
> Adding header 'sha-hash' I to 01178-81-3305-217
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01178-81-3305-217> stored/uncompressed,indexed<uplibdate:20070510>
> stored/uncompressed,indexed<uplibtype:whole>>
> Added 01178-81-3305-217 (1 versions)
> Working on document /local/janssen-uplib/docs/01178-73-1029-141
> Adding header 'apparent-mime-type' I to 01178-73-1029-141
> Adding header 'authors' IT to 01178-73-1029-141
> Adding header 'categories' I (article) to 01178-73-1029-141
> Adding header 'date' I (20070514) to 01178-73-1029-141
> Adding header 'sha-hash' I to 01178-73-1029-141
> Adding header 'title' IT (Critical Mass: Everyone listens to
> Walter Mossberg) to 01178-73-1029-141
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01178-73-1029-141> stored/uncompressed,indexed<uplibdate:20070509>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (328): Go Back Print this page Skip t
> page 1 (1024): Mossberg assesses technology p
> page 2 (4281): that Mossberg has since descri
> page 3 (2610): Titus said, ÒIt will come with
> page 4 (4412): ÒWeÕd love that,Ó Mermelstein
> page 5 (4035): half years, and then transferr
> page 6 (4952): partly to the clout of the new
> page 7 (5049): Mossberg will often be the fir
> page 8 (4330): Of the blogs that review produ
> page 9 (31): ¥ Del.icio.us ¥ Reddit Object
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01178-73-1029-141 (11 versions)
> Working on document /local/janssen-uplib/docs/01178-72-8365-803
> Adding header 'apparent-mime-type' I to 01178-72-8365-803
> Adding header 'authors' IT to 01178-72-8365-803
> Adding header 'categories' I (ebooks) to 01178-72-8365-803
> Adding header 'categories' I (article) to 01178-72-8365-803
> Adding header 'date' I (20061104) to 01178-72-8365-803
> Adding header 'sha-hash' I to 01178-72-8365-803
> Adding header 'title' IT (Selling Ebooks on the Web via MHTML) to
> 01178-72-8365-803
> Created empty doc Document<stored/uncompressed,indexed<id:
> 01178-72-8365-803> stored/uncompressed,indexed<uplibdate:20070509>
> stored/uncompressed,indexed<uplibtype:whole>>
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> page 0 (3499): To: ebook-community@yahoogroup
> page 1 (2904): the IETF world, to be the resp
> Using charset utf8 for contents.txt
> Using language en for contents.txt
> Added 01178-72-8365-803 (3 versions)
> Optimizing...
> IW 0 [main]: optimize: index now _21:c19686 _22:c92
> IW 0 [main]: flush: segment=_23 docStoreSegment=_23
> docStoreOffset=0 flushDocs=true flushDeletes=true
> flushDocStores=true numDocs=92 numBufDelTerms=6
> IW 0 [main]: index before flush _21:c19686 _22:c92
>
> closeDocStore: 2 files to flush to segment _23
>
> flush postings as segment _23 numDocs=92
> oldRAMSize=528092 newFlushedSize=263152 docs/MB=366.59 new/
> old=49.831%
> IW 0 [main]: flush 6 buffered deleted terms on 3 segments.
> flushed 92 deleted documents
> IW 0 [main]: checkpoint: wrote segments file "segments_48"
> IFD [main]: now checkpoint "segments_48" [3 segments ; isCommit =
> true]
> IFD [main]: deleteCommits: now remove commit "segments_47"
> IFD [main]: delete "segments_47"
> IW 0 [main]: checkpoint: wrote segments file "segments_49"
> IFD [main]: now checkpoint "segments_49" [3 segments ; isCommit =
> true]
> IFD [main]: deleteCommits: now remove commit "segments_48"
> IFD [main]: delete "_23.fnm"
> IFD [main]: delete "_23.frq"
> IFD [main]: delete "_23.prx"
> IFD [main]: delete "_23.tis"
> IFD [main]: delete "_23.tii"
> IFD [main]: delete "_23.nrm"
> IFD [main]: delete "_23.fdx"
> IFD [main]: delete "_23.fdt"
> IFD [main]: delete "segments_48"
> IW 0 [main]: LMP: findMerges: 3 segments
> IW 0 [main]: LMP: level 6.744677 to 7.494677: 1 segments
> IW 0 [main]: LMP: level -1.0 to 5.513348: 2 segments
> IW 0 [main]: CMS: now merge
> IW 0 [main]: CMS: index: _21:c19686 _22:c92 _23:c92
> IW 0 [main]: CMS: no more merges pending; now return
> IW 0 [main]: add merge to pendingMerges: _21:c19686 _22:c92 _23:c92
> [optimize] [total 1 pending]
> IW 0 [main]: CMS: now merge
> IW 0 [main]: CMS: index: _21:c19686 _22:c92 _23:c92
> IW 0 [main]: CMS: consider merge _21:c19686 _22:c92 _23:c92 into
> _24 [optimize]
> IW 0 [main]: CMS: launch new thread [Thread-0]
> IW 0 [Thread-0]: CMS: merge thread: start
> IW 0 [main]: CMS: no more merges pending; now return
> IW 0 [Thread-0]: now merge
> merge=_21:c19686 _22:c92 _23:c92 into _24 [optimize]
> index=_21:c19686 _22:c92 _23:c92
> IW 0 [Thread-0]: merging _21:c19686 _22:c92 _23:c92 into _24
> [optimize]
> IW 0 [Thread-0]: merge: total 19686 docs
> IW 0 [Thread-0]: hit exception during merge; now refresh deleter on
> segment _24
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.fdt"
> IFD [Thread-0]: delete "_24.fdt"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.fdx"
> IFD [Thread-0]: delete "_24.fdx"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.fnm"
> IFD [Thread-0]: delete "_24.fnm"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.frq"
> IFD [Thread-0]: delete "_24.frq"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.prx"
> IFD [Thread-0]: delete "_24.prx"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.tii"
> IFD [Thread-0]: delete "_24.tii"
> IFD [Thread-0]: refresh [prefix=_24]: removing newly created
> unreferenced file "_24.tis"
> IFD [Thread-0]: delete "_24.tis"
> IW 0 [Thread-0]: hit exception during merge
> Exception in thread "Thread-0" org.apache.lucene.index.MergePolicy
> $MergeException: java.lang.ArrayIndexOutOfBoundsException: Array
> index out of range: 20672
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:274)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out
> of range: 20672
> at org.apache.lucene.util.BitVector.get(BitVector.java:72)
> at
> org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
> at
> org
> .apache
> .lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
> at
> org
> .apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:
> 467)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:
> 430)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:
> 402)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:
> 366)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
> 123)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:
> 3002)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:240)
> java.io.IOException: background merge hit exception: _21:c19686
> _22:c92 _23:c92 into _24 [optimize]
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
> 1705)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
> 1654)
> at
> com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:419)
> at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:
> 664)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out
> of range: 20672
> at org.apache.lucene.util.BitVector.get(BitVector.java:72)
> at
> org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
> at
> org
> .apache
> .lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
> at
> org
> .apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:
> 467)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:
> 430)
> at
> org
> .apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:
> 402)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:
> 366)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
> 123)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:
> 3002)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:240)
> janssen-home : /u 75 %
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Can you try running with the trunk version of Lucene (2.3-dev) and see
> if the error still occurs? EG you can download this AM's build here:
>
> http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/288/artifact/artifacts
Still there. Here's the dump with last night's build:
/Library/Java/Home/bin/java '-Dcom.parc.uplib.indexing.debugMode=true' '-Dcom.parc.uplib.indexing.indexProperties=contents:title:categories$,*:date@:apparent-mime-type*:authors$\sand\s:comment:abstract:email-message-id*:email-guid*:email-subject:email-from-name:email-from-address*:email-attachment-to*:email-thread-index*:email-references$,*:email-in-reply-to$,*:keywords$,*:album:performer:composer:music-genre*:audio-length:accompaniment:paragraph-ids$,*:sha-hash*' -classpath "/local/uplib/share/UpLib-1.7/code/lucene-core-2.3-2007-11-29_02-49-31.jar:/local/uplib/share/UpLib-1.7/code/LuceneIndexing.jar" -Dorg.apache.lucene.writeLockTimeout=20000 com.parc.uplib.indexing.LuceneIndexing "/local/janssen-uplib/index" update /local/janssen-uplib/docs 01179-00-0750-547 01178-90-9186-558 01178-81-4212-772 01178-81-3305-217 01178-73-1029-141 01178-72-8365-803
updating
doc_root_dir is /local/janssen-uplib/docs
IFD [main]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@462851
IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/local/janssen-uplib/index autoCommit=true mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@c56c60 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@4e280c ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=_21:c19686 _22:c92
IW 0 [main]: setMaxFieldLength 2147483647
Working on document /local/janssen-uplib/docs/01179-00-0750-547
Adding header 'abstract' IT to 01179-00-0750-547
Adding header 'apparent-mime-type' I to 01179-00-0750-547
Adding header 'authors' IT to 01179-00-0750-547
Adding header 'categories' I (ebooks) to 01179-00-0750-547
Adding header 'categories' I (economics) to 01179-00-0750-547
Adding header 'categories' I (paper) to 01179-00-0750-547
Adding header 'citation' I to 01179-00-0750-547
Adding header 'date' I (20070128) to 01179-00-0750-547
Adding header 'sha-hash' I to 01179-00-0750-547
Adding header 'title' IT (Heterogeneity in Price Stickiness and the Real Effects of Monetary Shocks) to 01179-00-0750-547
Created empty doc Document<stored/uncompressed,indexed<id:01179-00-0750-547> stored/uncompressed,indexed<uplibdate:20070512> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (2181): Heterogeneity in Price Stickin
page 1 (2927): 1 Introduction There is ample
page 2 (3135): In the presence of strategic c
page 3 (3128): Motivated by those questions,
page 4 (3214): ploring the tractability of th
page 5 (2491): model with Taylor staggered wa
page 6 (1548): real rigidities (Ball and Rome
page 7 (3098): 2.2 Calibrating the sectoral d
page 8 (1913): distribution of price stickine
page 9 (1952): reported in Table 1. Hencefort
page 10 (1635): Figure 2 presents analogous re
page 11 (1743): In the absence of strategic co
page 12 (2806): Corollary 1 For an arbitrary h
page 13 (2380): 2.4.2 Growth rate shocks In th
page 14 (2962): price changes. With heterogene
page 15 (3265): ties and heterogeneity in the
page 16 (1962): complementarities. The results
page 17 (751): to the response of the heterog
page 18 (489): economies are embedded into th
page 19 (3295): 2.6 Fitting IRFs with an ident
page 20 (2066): Table 3a: Best-Fitting Duratio
page 21 (2444): This is an important step beca
page 22 (1976): where ? is the discount factor
page 23 (1183): Et "? Ct+1 Ct �?? It Pt Pt+1 #
page 24 (2188): can be rewritten as: Pk,t = �
page 25 (1370): pt = Z 1 0 f (k) pk,tdk, (10)
page 26 (3269): Heterogeneity in price stickin
page 27 (3117): Irrespective of the net effect
page 28 (2084): set of parameters involve high
page 29 (575): 0 5 10 15 20 25 30 35 40 0 x 1
page 30 (2185): output and falling prices in a
page 31 (2358): price changes that minimizes t
page 32 (2689): These results are fully consis
page 33 (3600): different sources of real rigi
page 34 (3168): work in a model with heterogen
page 35 (2557): single equation estimation of
page 36 (1326): Taking the limit as � ? 0 in e
page 37 (1796): The output gap is constant at
page 38 (1066): The corresponding path for the
page 39 (1347): 4) Proof of Corollaries 1 and
page 40 (2421): Therefore, for ? � 0, the expe
page 41 (1343): p (t) = Z 1 0 f (k) ? ?? ?? R
page 42 (2117): As ? ? 0, this clearly converg
page 43 (1375): model around the zero inflatio
page 44 (1497): pt = Z 1 0 f (k) pk,tdk, yt =
page 45 (1128): Table A.3: Best-Fitting Durati
page 46 (898): Multiplying by f (k) ?k and in
page 47 (1072): Now, from (23): ?kxk,t = pk,t
page 48 (268): Finally, let �t ? pt ? pt?1 de
page 49 (1694): References [1] Altissimo, F.,
page 50 (1874): [14] Bils, M., P. Klenow and O
page 51 (2091): [27] Carlton, D. (1986), �The
page 52 (1846): [39] Dixon, H. and E. Kara (20
page 53 (1530): [51] Ohanian, L., A. Stockman
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01179-00-0750-547 (55 versions)
Working on document /local/janssen-uplib/docs/01178-90-9186-558
Adding header 'abstract' IT to 01178-90-9186-558
Adding header 'apparent-mime-type' I to 01178-90-9186-558
Adding header 'authors' IT to 01178-90-9186-558
Adding header 'authors' IT to 01178-90-9186-558
Adding header 'authors' IT to 01178-90-9186-558
Adding header 'authors' IT to 01178-90-9186-558
Adding header 'authors' IT to 01178-90-9186-558
Adding header 'categories' I (paper) to 01178-90-9186-558
Adding header 'categories' I (ebooks) to 01178-90-9186-558
Adding header 'date' I (20050500) to 01178-90-9186-558
Adding header 'sha-hash' I to 01178-90-9186-558
Adding header 'title' IT (Visual-Syntactic Text Formatting: A New Method to Enhance Online) to 01178-90-9186-558
Created empty doc Document<stored/uncompressed,indexed<id:01178-90-9186-558> stored/uncompressed,indexed<uplibdate:20070511> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3332): Visual-Syntactic Text Formatti
page 1 (1461): into this: To make these chang
page 2 (4620): complex than a simple, concate
page 3 (4723): Among some poorer readers, met
page 4 (3827): method does not extract or dir
page 5 (3388): (ACT scores) with gains in com
page 6 (3693): digital text actually improve
page 7 (3400): exam were administered. For re
page 8 (3413): Intermediate and long-term ret
page 9 (4219): Student preference and survey
page 10 (4417): Discussion In print media, sim
page 11 (4358): time; however, the opposite tr
page 12 (3950): More time spent actually readi
page 13 (4184): In education, the VSTF method
page 14 (3270): References Armbruster, B.B. (2
page 15 (3176): April 19�20). Neuroimaging, la
page 16 (3350): Klare, G.R., Nichols, W.H., &
page 17 (4098): and narrative skills connect w
page 18 (3070): He has conducted laboratory-ba
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01178-90-9186-558 (20 versions)
Working on document /local/janssen-uplib/docs/01178-81-4212-772
Adding header 'abstract' IT to 01178-81-4212-772
Adding header 'apparent-mime-type' I to 01178-81-4212-772
Adding header 'categories' I (newspaper) to 01178-81-4212-772
Adding header 'categories' I (article) to 01178-81-4212-772
Adding header 'categories' I (fun) to 01178-81-4212-772
Adding header 'categories' I (historical) to 01178-81-4212-772
Adding header 'date' I (19340429) to 01178-81-4212-772
Adding header 'sha-hash' I to 01178-81-4212-772
Adding header 'title' IT (Gigantic Robots, Controlled by Wireless, to Fight Our Battles) to 01178-81-4212-772
Created empty doc Document<stored/uncompressed,indexed<id:01178-81-4212-772> stored/uncompressed,indexed<uplibdate:20070510> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (261): iganlic obok ntroIIed b ireles
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01178-81-4212-772 (2 versions)
Working on document /local/janssen-uplib/docs/01178-81-3305-217
Adding header 'apparent-mime-type' I to 01178-81-3305-217
Adding header 'authors' IT to 01178-81-3305-217
Adding header 'categories' I (cartoon) to 01178-81-3305-217
Adding header 'keywords' I (incentive) to 01178-81-3305-217
Adding header 'sha-hash' I to 01178-81-3305-217
Created empty doc Document<stored/uncompressed,indexed<id:01178-81-3305-217> stored/uncompressed,indexed<uplibdate:20070510> stored/uncompressed,indexed<uplibtype:whole>>
Added 01178-81-3305-217 (1 versions)
Working on document /local/janssen-uplib/docs/01178-73-1029-141
Adding header 'apparent-mime-type' I to 01178-73-1029-141
Adding header 'authors' IT to 01178-73-1029-141
Adding header 'categories' I (article) to 01178-73-1029-141
Adding header 'date' I (20070514) to 01178-73-1029-141
Adding header 'sha-hash' I to 01178-73-1029-141
Adding header 'title' IT (Critical Mass: Everyone listens to Walter Mossberg) to 01178-73-1029-141
Created empty doc Document<stored/uncompressed,indexed<id:01178-73-1029-141> stored/uncompressed,indexed<uplibdate:20070509> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (328): Go Back Print this page Skip t
page 1 (1024): Mossberg assesses technology p
page 2 (4281): that Mossberg has since descri
page 3 (2610): Titus said, �It will come with
page 4 (4412): �We�d love that,� Mermelstein
page 5 (4035): half years, and then transferr
page 6 (4952): partly to the clout of the new
page 7 (5049): Mossberg will often be the fir
page 8 (4330): Of the blogs that review produ
page 9 (31): � Del.icio.us � Reddit Object
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01178-73-1029-141 (11 versions)
Working on document /local/janssen-uplib/docs/01178-72-8365-803
Adding header 'apparent-mime-type' I to 01178-72-8365-803
Adding header 'authors' IT to 01178-72-8365-803
Adding header 'categories' I (ebooks) to 01178-72-8365-803
Adding header 'categories' I (article) to 01178-72-8365-803
Adding header 'date' I (20061104) to 01178-72-8365-803
Adding header 'sha-hash' I to 01178-72-8365-803
Adding header 'title' IT (Selling Ebooks on the Web via MHTML) to 01178-72-8365-803
Created empty doc Document<stored/uncompressed,indexed<id:01178-72-8365-803> stored/uncompressed,indexed<uplibdate:20070509> stored/uncompressed,indexed<uplibtype:whole>>
Using charset utf8 for contents.txt
Using language en for contents.txt
page 0 (3499): To: ebook-community@yahoogroup
page 1 (2904): the IETF world, to be the resp
Using charset utf8 for contents.txt
Using language en for contents.txt
Added 01178-72-8365-803 (3 versions)
Optimizing...
IW 0 [main]: optimize: index now _21:c19686 _22:c92
IW 0 [main]: flush: segment=_23 docStoreSegment=_23 docStoreOffset=0 flushDocs=true flushDeletes=true flushDocStores=true numDocs=92 numBufDelTerms=6
IW 0 [main]: index before flush _21:c19686 _22:c92
closeDocStore: 2 files to flush to segment _23
flush postings as segment _23 numDocs=92
oldRAMSize=528092 newFlushedSize=263152 docs/MB=366.59 new/old=49.831%
IW 0 [main]: flush 6 buffered deleted terms on 3 segments.
flushed 92 deleted documents
IW 0 [main]: checkpoint: wrote segments file "segments_48"
IFD [main]: now checkpoint "segments_48" [3 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_47"
IFD [main]: delete "segments_47"
IW 0 [main]: checkpoint: wrote segments file "segments_49"
IFD [main]: now checkpoint "segments_49" [3 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_48"
IFD [main]: delete "_23.fnm"
IFD [main]: delete "_23.frq"
IFD [main]: delete "_23.prx"
IFD [main]: delete "_23.tis"
IFD [main]: delete "_23.tii"
IFD [main]: delete "_23.nrm"
IFD [main]: delete "_23.fdx"
IFD [main]: delete "_23.fdt"
IFD [main]: delete "segments_48"
IW 0 [main]: LMP: findMerges: 3 segments
IW 0 [main]: LMP: level 6.744677 to 7.494677: 1 segments
IW 0 [main]: LMP: level -1.0 to 5.513348: 2 segments
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _21:c19686 _22:c92 _23:c92
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [main]: add merge to pendingMerges: _21:c19686 _22:c92 _23:c92 [optimize] [total 1 pending]
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _21:c19686 _22:c92 _23:c92
IW 0 [main]: CMS: consider merge _21:c19686 _22:c92 _23:c92 into _24 [optimize]
IW 0 [main]: CMS: launch new thread [Thread-0]
IW 0 [Thread-0]: CMS: merge thread: start
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [Thread-0]: now merge
merge=_21:c19686 _22:c92 _23:c92 into _24 [optimize]
index=_21:c19686 _22:c92 _23:c92
IW 0 [Thread-0]: merging _21:c19686 _22:c92 _23:c92 into _24 [optimize]
IW 0 [Thread-0]: merge: total 19686 docs
IW 0 [Thread-0]: hit exception during merge; now refresh deleter on segment _24
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.fdt"
IFD [Thread-0]: delete "_24.fdt"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.fdx"
IFD [Thread-0]: delete "_24.fdx"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.fnm"
IFD [Thread-0]: delete "_24.fnm"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.frq"
IFD [Thread-0]: delete "_24.frq"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.prx"
IFD [Thread-0]: delete "_24.prx"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.tii"
IFD [Thread-0]: delete "_24.tii"
IFD [Thread-0]: refresh [prefix=_24]: removing newly created unreferenced file "_24.tis"
IFD [Thread-0]: delete "_24.tis"
IW 0 [Thread-0]: hit exception during merge
Exception in thread "Thread-0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 20672
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:274)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 20672
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:467)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
java.io.IOException: background merge hit exception: _21:c19686 _22:c92 _23:c92 into _24 [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1705)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1654)
at com.parc.uplib.indexing.LuceneIndexing.update(LuceneIndexing.java:419)
at com.parc.uplib.indexing.LuceneIndexing.main(LuceneIndexing.java:664)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 20672
at org.apache.lucene.util.BitVector.get(BitVector.java:72)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:118)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:95)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:467)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:430)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:402)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:366)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3002)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2751)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
janssen-home : /u 75 %
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Bill Janssen" <ja...@parc.com> wrote:
> > Hmmm ... how many chunks of "about 50 pages" do you do before
> > hitting this? Roughly how many docs are in the index when it
> > happens?
>
> Oh, gosh, not sure. I'm guessing it's about half done.
Ugh, OK. If we could boil this down to a smaller set that is easily
reproducible (and transferable to me) then I could try to track it
down.
Do you have another PPC machine to reproduce this on? (To rule out
bad RAM/hard-drive on the first one).
Can you try running with the trunk version of Lucene (2.3-dev) and see
if the error still occurs? EG you can download this AM's build here:
http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/288/artifact/artifacts
Another thing to try is turning on the infoStream
(IndexWriter.setInfoStream(...)) and capture & post the resulting log.
It will be very large since it takes quite a while for the error to
occur...
> So, I ran the same codebase with lucene-core-2.2.0.jar on an Intel
> Mac Pro, OS X 10.5.0, Java 1.5, and no exception is raised.
> Different corpus, about 50000 pages instead of 20000. This is
> reinforcing my thinking that it's a big-endian issue.
That's a good question.
Lucene is endian independent: all writes to files boil eventually down
to a writeByte/writeBytes calls in o.a.l.store.IndexOutput such that
the ordering is controlled by Lucene, not the underlying CPU
architecture.
That said, it is clearly a difference in your test so it seems like a
compelling lead... is it possible to run this different corpus back on
the PPC machine, to rule out a corpus difference leading to the
exception?
> I've got 1735 documents, 18969 pages -- average page size 10.9, max
> page size 1235 (a physics textbook), 578 one-page documents. These
> are Web pages, PDFs, articles, photos, scanned stuff, technical
> papers, etc. I index six documents at a time, so I guess I'm
> averaging about 65 pages per chunk. For each document, I index the
> whole text of the document as a Lucene Document, and I index the
> text of each page separately as a Document. I use the "contents"
> fields and "pagecontents" fields for those two uses. I also add
> metadata information to each: "title", multiple "author" fields,
> "date", "abstract", etc.
OK, sounds like a nice rich corpus :) Are you using term vectors,
stored fields, payloads on any of these?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Hmmm ... how many chunks of "about 50 pages" do you do before hitting this?
> Roughly how many docs are in the index when it happens?
Oh, gosh, not sure. I'm guessing it's about half done.
> Can you describe the docs/fields you're adding?
I've got 1735 documents, 18969 pages -- average page size 10.9, max
page size 1235 (a physics textbook), 578 one-page documents. These
are Web pages, PDFs, articles, photos, scanned stuff, technical
papers, etc. I index six documents at a time, so I guess I'm
averaging about 65 pages per chunk. For each document, I index the
whole text of the document as a Lucene Document, and I index the text
of each page separately as a Document. I use the "contents" fields
and "pagecontents" fields for those two uses. I also add metadata
information to each: "title", multiple "author" fields, "date",
"abstract", etc.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmmm ... how many chunks of "about 50 pages" do you do before hitting this?
Roughly how many docs are in the index when it happens?
Can you describe the docs/fields you're adding?
You are not hitting any other exception before this one right?
Can you change your test case so that the "catch" clause is run
before the "finally" clause? I wonder if you are hitting some
interesting exception and then trying to optimize, which then
masks the original exception.
Mike
"Bill Janssen" <ja...@parc.com> wrote:
> > Are you really sure in your 2.2 test you are starting with no prior
> > index?
>
> I'd ask that too, but yes, I'm really really sure. Building a
> completely new index each time.
>
> Works with 2.0.0. Fails with 2.2.0. Works with 2.2.0 *if* I remove
> the optimization step.
>
> Bill
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
> Are you really sure in your 2.2 test you are starting with no prior
> index?
I'd ask that too, but yes, I'm really really sure. Building a
completely new index each time.
Works with 2.0.0. Fails with 2.2.0. Works with 2.2.0 *if* I remove
the optimization step.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Are you really sure in your 2.2 test you are starting with no prior
index?
2.2 should in fact work fine with a 2.0 index but it's possible there
was some latent corruption in the 2.0 index if you are accidentally
using it. That exception looks alot like this dreaded bug:
https://issues.apache.org/jira/browse/LUCENE-140
which was fixed in 2.1.
Mike
"Bill Janssen" <ja...@parc.com> wrote:
> I just tried re-indexing with lucene-core-2.0.0.jar and the same
> indexing code; works great. So what am I doing wrong with 2.2?
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene-core-2.2.0.jar broken? CorruptIndexException?
Posted by Bill Janssen <ja...@parc.com>.
I just tried re-indexing with lucene-core-2.0.0.jar and the same
indexing code; works great. So what am I doing wrong with 2.2?
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org