You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/09/08 17:17:35 UTC
[jira] Updated: (LUCENE-2637) FSDirectory.copyBytes isn't safe for
SimpleFSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2637:
--------------------------------
Summary: FSDirectory.copyBytes isn't safe for SimpleFSDirectory (was: TestIndexWriter.testRandomStoredFields randomly fails)
Affects Version/s: 3.1
Description:
the copyBytes optimization from LUCENE-2574 is not safe for SimpleFSDirectory, but works fine for NIOFSDirectory.
With SimpleFSDirectory, the copyBytes optimization causes index corruption.
see http://www.lucidimagination.com/search/document/36d2dbfc691909d5/bug_triggered_by_testindexwriter_testrandomstoredfields for background
here are my steps to reproduce (most of the time, at least on windows):
{noformat}
1. edit line 87 of TestIndexWriter to plugin the seed:
random = newRandom(3312389322103990899L);
2. edit line 5138 of TestIndexWriter to force SimpleFSDirectory:
Directory dir = new SimpleFSDirectory(index);
3. run this command:
ant clean test-core -Dtestcase=TestIndexWriter
-Dtestmethod=testRandomStoredFields -Dtests.iter=10
-Dtests.codec="MockVariableIntBlock(29)"
{noformat}
was:
see http://www.lucidimagination.com/search/document/36d2dbfc691909d5/bug_triggered_by_testindexwriter_testrandomstoredfields for background
here are my steps to reproduce (most of the time, at least on windows):
{noformat}
1. edit line 87 of TestIndexWriter to plugin the seed:
random = newRandom(3312389322103990899L);
2. run this command:
ant clean test-core -Dtestcase=TestIndexWriter
-Dtestmethod=testRandomStoredFields -Dtests.iter=10
-Dtests.codec="MockVariableIntBlock(29)"
{noformat}
I only set this up on trunk, it might affect 3x too (I feel like i have seen it, but i'm not sure and i didnt test yet)
here are a few of the various types of fails i have seen:
{noformat}
[junit] Testsuite: org.apache.lucene.index.TestIndexWriter
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): Caused an ERROR
[junit] Index: 4232, Size: 4
[junit] java.lang.IndexOutOfBoundsException: Index: 4232, Size: 4
[junit] at java.util.ArrayList.RangeCheck(ArrayList.java:547)
[junit] at java.util.ArrayList.get(ArrayList.java:322)
[junit] at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:264)
[junit] at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:198)
[junit] at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:882)
[junit] at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:580)
[junit] at org.apache.lucene.index.IndexReader.document(IndexReader.java:754)
[junit] at org.apache.lucene.index.TestIndexWriter.testRandomStoredFields(TestIndexWriter.java:5212)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:397)
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:389)
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): Caused an ERROR
[junit] null
[junit] java.lang.NullPointerException
[junit] at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:310)
[junit] at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:209)
[junit] at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:882)
[junit] at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:580)
[junit] at org.apache.lucene.index.IndexReader.document(IndexReader.java:754)
[junit] at org.apache.lucene.index.TestIndexWriter.testRandomStoredFields(TestIndexWriter.java:5212)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:397)
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:389)
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:389)
[junit] at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:202)
[junit] at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:882)
[junit] at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:580)
[junit] at org.apache.lucene.index.IndexReader.document(IndexReader.java:754)
[junit] at org.apache.lucene.index.TestIndexWriter.testRandomStoredFields(TestIndexWriter.java:5212)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:397)
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): FAILED
[junit] doc 16, field f3 is wrong expected:<nonsense unicode> but was <different nonsense unicode>
[junit] junit.framework.ComparisonFailure: doc 16, field f3 is wrong expected: <nonsense unicode>
but was <different nonsense unicode>
[junit] at org.apache.lucene.index.TestIndexWriter.testRandomStoredFields(TestIndexWriter.java:5215)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:397)
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:389)
[junit]
[junit] Testsuite: org.apache.lucene.index.TestIndexWriter
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): Caused an ERROR
[junit] null
[junit] java.lang.NegativeArraySizeException
[junit] at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:305)
[junit] at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:209)
[junit] at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:882)
[junit] at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:580)
[junit] at org.apache.lucene.index.IndexReader.document(IndexReader.java:754)
[junit] at org.apache.lucene.index.TestIndexWriter.testRandomStoredFields(TestIndexWriter.java:5212)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:397)
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:389)
[junit] Testsuite: org.apache.lucene.index.TestIndexWriter
[junit] Testcase: testRandomStoredFields(org.apache.lucene.index.TestIndexWriter): Caused an ERROR
[junit] read past EOF
[junit] java.io.IOException: read past EOF
[junit] at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:138)
...
{noformat}
editing the issue title/description now that we understand the problem.
> FSDirectory.copyBytes isn't safe for SimpleFSDirectory
> ------------------------------------------------------
>
> Key: LUCENE-2637
> URL: https://issues.apache.org/jira/browse/LUCENE-2637
> Project: Lucene - Java
> Issue Type: Bug
> Components: Tests
> Affects Versions: 3.1, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-2637.patch
>
>
> the copyBytes optimization from LUCENE-2574 is not safe for SimpleFSDirectory, but works fine for NIOFSDirectory.
> With SimpleFSDirectory, the copyBytes optimization causes index corruption.
> see http://www.lucidimagination.com/search/document/36d2dbfc691909d5/bug_triggered_by_testindexwriter_testrandomstoredfields for background
> here are my steps to reproduce (most of the time, at least on windows):
> {noformat}
> 1. edit line 87 of TestIndexWriter to plugin the seed:
> random = newRandom(3312389322103990899L);
> 2. edit line 5138 of TestIndexWriter to force SimpleFSDirectory:
> Directory dir = new SimpleFSDirectory(index);
> 3. run this command:
> ant clean test-core -Dtestcase=TestIndexWriter
> -Dtestmethod=testRandomStoredFields -Dtests.iter=10
> -Dtests.codec="MockVariableIntBlock(29)"
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org