You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2009/05/31 22:31:07 UTC
[jira] Created: (LUCENE-1669) MMapDirectory on Windows silently
fails to write to a file if also open for read
MMapDirectory on Windows silently fails to write to a file if also open for read
--------------------------------------------------------------------------------
Key: LUCENE-1669
URL: https://issues.apache.org/jira/browse/LUCENE-1669
Project: Lucene - Java
Issue Type: Bug
Components: Store
Reporter: Michael McCandless
Priority: Minor
This is not a normal situation Lucene currently encounters, though we
are discussing exactly this possibility in LUCENE-1313.
I only hit it in digging down on a test failure in LUCENE-1658:
{code}
[junit] Testcase: testIndexAndMerge(org.apache.lucene.index.TestDoc): FAILED
[junit] junit.framework.AssertionFailedError:
[junit] at org.apache.lucene.index.FieldsWriter.addRawDocuments(FieldsWriter.java:249)
[junit] at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:350)
[junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
[junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
[junit] at org.apache.lucene.index.TestDoc.merge(TestDoc.java:182)
[junit] at org.apache.lucene.index.TestDoc.testIndexAndMerge(TestDoc.java:117)
[junit] at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
{code}
That failure happens on Windows 64bit, if you use MMapDirectory, and
the index is on a remote (CIFS) mount.
This test opens a SegmentReader against doc store files that
IndexWriter still has open.
I whittled it down to this test showing the root cause:
{code}
public void testMMapWriteRead() throws Exception {
MMapDirectory dir = new MMapDirectory(new File("readwrite"), null);
//NIOFSDirectory dir = new NIOFSDirectory(new File("readwrite"), null);
//SimpleFSDirectory dir = new SimpleFSDirectory(new File("readwrite"), null);
IndexOutput out = dir.createOutput("one");
out.writeLong(17);
out.flush();
// open the same file we are writing
dir.openInput("one").close();
// write another long
out.writeLong(56);
out.close();
IndexInput in = dir.openInput("one");
assertEquals(17, in.readLong());
assertEquals(56, in.readLong());
in.close();
}
{code}
NIOFSDir and SimpleFSDir pass the test fine, which is nice to know
(for LUCENE-1313). MMapDir passes fine on a local drive, but fails on
a remote CIFS mount.
I'm not sure what's going on. It seems like it could be a JRE bug --
behavior shouldn't change on local vs remote drive. I've only tested
it with java 1.6.0_11.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1669) MMapDirectory on Windows silently
fails to write to a file if also open for read
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714927#action_12714927 ]
Uwe Schindler commented on LUCENE-1669:
---------------------------------------
In the mmap bug report (we know it) at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038 one user states:
{quote}
Submitted On 13-MAR-2006
sebastien.vauclair
I am using the workaround shown bellow to unmap buffers.
Everything works fine with multiple applications reading from the same growing local file, but there is a problem if the file is accessed using a Windows UNC path like "//machine/c$/dir/file": Buffer.get() returns 0 for some bytes near EOF in such files.
This bug is reproductible even with 127.0.0.1 as the machine.
Should I open a new bug? I was not sure considering the workaround used to unmap has not been officially approved.
Note: adding a 1 second sleep after the buffer has been GC'd fixes the problem - but that's of course not always acceptable.
{quote}
Maybe this is the cause?
> MMapDirectory on Windows silently fails to write to a file if also open for read
> --------------------------------------------------------------------------------
>
> Key: LUCENE-1669
> URL: https://issues.apache.org/jira/browse/LUCENE-1669
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Reporter: Michael McCandless
> Priority: Minor
>
> This is not a normal situation Lucene currently encounters, though we
> are discussing exactly this possibility in LUCENE-1313.
> I only hit it in digging down on a test failure in LUCENE-1658:
> {code}
> [junit] Testcase: testIndexAndMerge(org.apache.lucene.index.TestDoc): FAILED
> [junit] junit.framework.AssertionFailedError:
> [junit] at org.apache.lucene.index.FieldsWriter.addRawDocuments(FieldsWriter.java:249)
> [junit] at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:350)
> [junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
> [junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
> [junit] at org.apache.lucene.index.TestDoc.merge(TestDoc.java:182)
> [junit] at org.apache.lucene.index.TestDoc.testIndexAndMerge(TestDoc.java:117)
> [junit] at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
> {code}
> That failure happens on Windows 64bit, if you use MMapDirectory, and
> the index is on a remote (CIFS) mount.
> This test opens a SegmentReader against doc store files that
> IndexWriter still has open.
> I whittled it down to this test showing the root cause:
> {code}
> public void testMMapWriteRead() throws Exception {
> MMapDirectory dir = new MMapDirectory(new File("readwrite"), null);
> //NIOFSDirectory dir = new NIOFSDirectory(new File("readwrite"), null);
> //SimpleFSDirectory dir = new SimpleFSDirectory(new File("readwrite"), null);
> IndexOutput out = dir.createOutput("one");
> out.writeLong(17);
> out.flush();
> // open the same file we are writing
> dir.openInput("one").close();
> // write another long
> out.writeLong(56);
> out.close();
> IndexInput in = dir.openInput("one");
> assertEquals(17, in.readLong());
> assertEquals(56, in.readLong());
> in.close();
> }
> {code}
> NIOFSDir and SimpleFSDir pass the test fine, which is nice to know
> (for LUCENE-1313). MMapDir passes fine on a local drive, but fails on
> a remote CIFS mount.
> I'm not sure what's going on. It seems like it could be a JRE bug --
> behavior shouldn't change on local vs remote drive. I've only tested
> it with java 1.6.0_11.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1669) MMapDirectory on Windows silently
fails to write to a file if also open for read
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714947#action_12714947 ]
Michael McCandless commented on LUCENE-1669:
--------------------------------------------
That sounds exactly like what I was hitting -- getting 0 from readLong instead of the expected 56, in the above test. Except, my test fails even before doing the hack... so it's not the hack that's causing it.
> MMapDirectory on Windows silently fails to write to a file if also open for read
> --------------------------------------------------------------------------------
>
> Key: LUCENE-1669
> URL: https://issues.apache.org/jira/browse/LUCENE-1669
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Reporter: Michael McCandless
> Priority: Minor
>
> This is not a normal situation Lucene currently encounters, though we
> are discussing exactly this possibility in LUCENE-1313.
> I only hit it in digging down on a test failure in LUCENE-1658:
> {code}
> [junit] Testcase: testIndexAndMerge(org.apache.lucene.index.TestDoc): FAILED
> [junit] junit.framework.AssertionFailedError:
> [junit] at org.apache.lucene.index.FieldsWriter.addRawDocuments(FieldsWriter.java:249)
> [junit] at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:350)
> [junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
> [junit] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
> [junit] at org.apache.lucene.index.TestDoc.merge(TestDoc.java:182)
> [junit] at org.apache.lucene.index.TestDoc.testIndexAndMerge(TestDoc.java:117)
> [junit] at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
> {code}
> That failure happens on Windows 64bit, if you use MMapDirectory, and
> the index is on a remote (CIFS) mount.
> This test opens a SegmentReader against doc store files that
> IndexWriter still has open.
> I whittled it down to this test showing the root cause:
> {code}
> public void testMMapWriteRead() throws Exception {
> MMapDirectory dir = new MMapDirectory(new File("readwrite"), null);
> //NIOFSDirectory dir = new NIOFSDirectory(new File("readwrite"), null);
> //SimpleFSDirectory dir = new SimpleFSDirectory(new File("readwrite"), null);
> IndexOutput out = dir.createOutput("one");
> out.writeLong(17);
> out.flush();
> // open the same file we are writing
> dir.openInput("one").close();
> // write another long
> out.writeLong(56);
> out.close();
> IndexInput in = dir.openInput("one");
> assertEquals(17, in.readLong());
> assertEquals(56, in.readLong());
> in.close();
> }
> {code}
> NIOFSDir and SimpleFSDir pass the test fine, which is nice to know
> (for LUCENE-1313). MMapDir passes fine on a local drive, but fails on
> a remote CIFS mount.
> I'm not sure what's going on. It seems like it could be a JRE bug --
> behavior shouldn't change on local vs remote drive. I've only tested
> it with java 1.6.0_11.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org