You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "Digy (JIRA)" <ji...@apache.org> on 2008/07/18 17:28:31 UTC
[jira] Created: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Race condition in DocumentsWriter.UpdateDocument
------------------------------------------------
Key: LUCENENET-135
URL: https://issues.apache.org/jira/browse/LUCENENET-135
Project: Lucene.Net
Issue Type: Bug
Environment: 2.3.1
Reporter: Digy
There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
(testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
I am working on it.
DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-135) Race condition in DocumentsWriter.UpdateDocument
Posted by Digy <di...@gmail.com>.
Sorry, I replied to wrong eMail......
"Index was outside of the bounds of the array" exception is thrown from
DocumentsWriter.ByteBlockPool.AllocSlice. But I suspect from a bug
elsewhere.
DIGY
-----Original Message-----
From: Digy [mailto:digydigy@gmail.com]
Sent: Friday, August 08, 2008 10:57 PM
To: 'lucene-net-dev@incubator.apache.org'
Subject: RE: [jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
I think, SuportClass.patch is correct but revealed a hidden bug in
DocumentsWriter.
(Last comment in https://issues.apache.org/jira/browse/LUCENENET-135)
DIGY
-----Original Message-----
From: Doug Sale [mailto:dougsale@gmail.com]
Sent: Friday, August 08, 2008 9:52 PM
To: lucene-net-dev@incubator.apache.org
Subject: Re: [jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
After applying the SuportClass.patch (prior to the commits to HEAD
yesterday), IndexWriter.TestDocumentsWriterExceptionThreads() passed and
IndexWriter.TestDiverseDocs() failed. Additionally,
TestPayloads.TestThreadSafety() passed (which was failing before). However,
there were still issues with TestThreadSafety(), as evidenced by all the
exceptions being thrown.
I will check out the status of the SupportClass.patch with the latest HEAD
shortly.
On Fri, Aug 8, 2008 at 12:29 PM, Digy (JIRA) <ji...@apache.org> wrote:
>
> [
>
https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.
plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620982#acti
on_12620982]
>
> Digy commented on LUCENENET-135:
> --------------------------------
>
> Hi,
>
> 1) SupportClass.patch fixes
> "Index/TestIndexWriter/TestDocumentsWriterExceptionThreads"
> 2) In fact , There is no race condition in DocumentsWriter(at least for
> now). It all started when I saw two threads entering the "GetThreadState"
at
> the same time which is a synchronized method. So, I falsely determined
that
> it was a race condition. And finally, It turned out to be a bug in
> ThreadClass.
>
> ThreadClass'es static "Current" method always returns a new instance of
> ThreadClass,
> and because it is used as a key to HashTable
> "threadBindings"(DocumentsWriter.GetThreadState),
> {code}
> ThreadState state = (ThreadState)
> threadBindings[SupportClass.ThreadClass.Current()];
> if (state == null)
> {
> ...
> threadBindings[SupportClass.ThreadClass.Current()] = state;
> }
> {code}
> "State==null" never gets false and ,everytime, initializations of the
> first-time call are made.
>
> I think the patch is correct. But after applying that patch the following
> simple test case fails with an exception "Index was outside the bounds of
> the array".
> {code}
> [Test]
> public void TestIndexWriter1()
> {
> RAMDirectory dir = new RAMDirectory();
> IndexWriter wr = new IndexWriter(dir,new WhitespaceAnalyzer(),
> true);
> wr.SetRAMBufferSizeMB(0.1);
>
> Document doc = new Document();
> Field f1 = new Field("field1", "", Field.Store.YES,
> Field.Index.TOKENIZED);
> doc.Add(f1);
>
> for (int i = 0; i < 200000; i++)
> {
> f1.SetValue("some text, some more text,some text, some more
> text,some text, some more text,");
>
> try
> {
> wr.AddDocument(doc);
> //if (i % 100 == 0) wr.Flush(); //Make that pass
> }
> catch (Exception ex)
> {
> Console.WriteLine("loop:" + i + "\n" + ex.Message +
"\n"
> + ex.StackTrace);
> throw ex;
> }
>
> }
> wr.Close();
> }
> {code}
> *Any Idea?*
>
> PS1:*Uncommenting Flush statement in the code makes that test pass*
>
> PS2: If you decide to close this issue, please take a look at the
> DocumentsWriter.patch(which has nothing to do with any test case but
applied
> to Lucene.java after the release of 2.3.1)
>
>
> DIGY
>
> > Race condition in DocumentsWriter.UpdateDocument
> > ------------------------------------------------
> >
> > Key: LUCENENET-135
> > URL: https://issues.apache.org/jira/browse/LUCENENET-135
> > Project: Lucene.Net
> > Issue Type: Bug
> > Environment: 2.3.1
> > Reporter: Digy
> > Attachments: DocumentsWriter+SupportClass.rar,
> DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch,
> SupportClass.patch, SupportClass.patch
> >
> >
> > There is a race condition in DocumentsWriter.UpdateDocument. Locking the
> whole method solves the problem but this is not what is intended in java
> version
> > (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> > I am working on it.
> > DIGY
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
RE: [jira] Commented: (LUCENENET-135) Race condition in DocumentsWriter.UpdateDocument
Posted by Digy <di...@gmail.com>.
I think, SuportClass.patch is correct but revealed a hidden bug in
DocumentsWriter.
(Last comment in https://issues.apache.org/jira/browse/LUCENENET-135)
DIGY
-----Original Message-----
From: Doug Sale [mailto:dougsale@gmail.com]
Sent: Friday, August 08, 2008 9:52 PM
To: lucene-net-dev@incubator.apache.org
Subject: Re: [jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
After applying the SuportClass.patch (prior to the commits to HEAD
yesterday), IndexWriter.TestDocumentsWriterExceptionThreads() passed and
IndexWriter.TestDiverseDocs() failed. Additionally,
TestPayloads.TestThreadSafety() passed (which was failing before). However,
there were still issues with TestThreadSafety(), as evidenced by all the
exceptions being thrown.
I will check out the status of the SupportClass.patch with the latest HEAD
shortly.
On Fri, Aug 8, 2008 at 12:29 PM, Digy (JIRA) <ji...@apache.org> wrote:
>
> [
>
https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.
plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620982#acti
on_12620982]
>
> Digy commented on LUCENENET-135:
> --------------------------------
>
> Hi,
>
> 1) SupportClass.patch fixes
> "Index/TestIndexWriter/TestDocumentsWriterExceptionThreads"
> 2) In fact , There is no race condition in DocumentsWriter(at least for
> now). It all started when I saw two threads entering the "GetThreadState"
at
> the same time which is a synchronized method. So, I falsely determined
that
> it was a race condition. And finally, It turned out to be a bug in
> ThreadClass.
>
> ThreadClass'es static "Current" method always returns a new instance of
> ThreadClass,
> and because it is used as a key to HashTable
> "threadBindings"(DocumentsWriter.GetThreadState),
> {code}
> ThreadState state = (ThreadState)
> threadBindings[SupportClass.ThreadClass.Current()];
> if (state == null)
> {
> ...
> threadBindings[SupportClass.ThreadClass.Current()] = state;
> }
> {code}
> "State==null" never gets false and ,everytime, initializations of the
> first-time call are made.
>
> I think the patch is correct. But after applying that patch the following
> simple test case fails with an exception "Index was outside the bounds of
> the array".
> {code}
> [Test]
> public void TestIndexWriter1()
> {
> RAMDirectory dir = new RAMDirectory();
> IndexWriter wr = new IndexWriter(dir,new WhitespaceAnalyzer(),
> true);
> wr.SetRAMBufferSizeMB(0.1);
>
> Document doc = new Document();
> Field f1 = new Field("field1", "", Field.Store.YES,
> Field.Index.TOKENIZED);
> doc.Add(f1);
>
> for (int i = 0; i < 200000; i++)
> {
> f1.SetValue("some text, some more text,some text, some more
> text,some text, some more text,");
>
> try
> {
> wr.AddDocument(doc);
> //if (i % 100 == 0) wr.Flush(); //Make that pass
> }
> catch (Exception ex)
> {
> Console.WriteLine("loop:" + i + "\n" + ex.Message +
"\n"
> + ex.StackTrace);
> throw ex;
> }
>
> }
> wr.Close();
> }
> {code}
> *Any Idea?*
>
> PS1:*Uncommenting Flush statement in the code makes that test pass*
>
> PS2: If you decide to close this issue, please take a look at the
> DocumentsWriter.patch(which has nothing to do with any test case but
applied
> to Lucene.java after the release of 2.3.1)
>
>
> DIGY
>
> > Race condition in DocumentsWriter.UpdateDocument
> > ------------------------------------------------
> >
> > Key: LUCENENET-135
> > URL: https://issues.apache.org/jira/browse/LUCENENET-135
> > Project: Lucene.Net
> > Issue Type: Bug
> > Environment: 2.3.1
> > Reporter: Digy
> > Attachments: DocumentsWriter+SupportClass.rar,
> DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch,
> SupportClass.patch, SupportClass.patch
> >
> >
> > There is a race condition in DocumentsWriter.UpdateDocument. Locking the
> whole method solves the problem but this is not what is intended in java
> version
> > (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> > I am working on it.
> > DIGY
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
Re: [jira] Commented: (LUCENENET-135) Race condition in DocumentsWriter.UpdateDocument
Posted by Doug Sale <do...@gmail.com>.
After applying the SuportClass.patch (prior to the commits to HEAD
yesterday), IndexWriter.TestDocumentsWriterExceptionThreads() passed and
IndexWriter.TestDiverseDocs() failed. Additionally,
TestPayloads.TestThreadSafety() passed (which was failing before). However,
there were still issues with TestThreadSafety(), as evidenced by all the
exceptions being thrown.
I will check out the status of the SupportClass.patch with the latest HEAD
shortly.
On Fri, Aug 8, 2008 at 12:29 PM, Digy (JIRA) <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620982#action_12620982]
>
> Digy commented on LUCENENET-135:
> --------------------------------
>
> Hi,
>
> 1) SupportClass.patch fixes
> "Index/TestIndexWriter/TestDocumentsWriterExceptionThreads"
> 2) In fact , There is no race condition in DocumentsWriter(at least for
> now). It all started when I saw two threads entering the "GetThreadState" at
> the same time which is a synchronized method. So, I falsely determined that
> it was a race condition. And finally, It turned out to be a bug in
> ThreadClass.
>
> ThreadClass'es static "Current" method always returns a new instance of
> ThreadClass,
> and because it is used as a key to HashTable
> "threadBindings"(DocumentsWriter.GetThreadState),
> {code}
> ThreadState state = (ThreadState)
> threadBindings[SupportClass.ThreadClass.Current()];
> if (state == null)
> {
> ...
> threadBindings[SupportClass.ThreadClass.Current()] = state;
> }
> {code}
> "State==null" never gets false and ,everytime, initializations of the
> first-time call are made.
>
> I think the patch is correct. But after applying that patch the following
> simple test case fails with an exception "Index was outside the bounds of
> the array".
> {code}
> [Test]
> public void TestIndexWriter1()
> {
> RAMDirectory dir = new RAMDirectory();
> IndexWriter wr = new IndexWriter(dir,new WhitespaceAnalyzer(),
> true);
> wr.SetRAMBufferSizeMB(0.1);
>
> Document doc = new Document();
> Field f1 = new Field("field1", "", Field.Store.YES,
> Field.Index.TOKENIZED);
> doc.Add(f1);
>
> for (int i = 0; i < 200000; i++)
> {
> f1.SetValue("some text, some more text,some text, some more
> text,some text, some more text,");
>
> try
> {
> wr.AddDocument(doc);
> //if (i % 100 == 0) wr.Flush(); //Make that pass
> }
> catch (Exception ex)
> {
> Console.WriteLine("loop:" + i + "\n" + ex.Message + "\n"
> + ex.StackTrace);
> throw ex;
> }
>
> }
> wr.Close();
> }
> {code}
> *Any Idea?*
>
> PS1:*Uncommenting Flush statement in the code makes that test pass*
>
> PS2: If you decide to close this issue, please take a look at the
> DocumentsWriter.patch(which has nothing to do with any test case but applied
> to Lucene.java after the release of 2.3.1)
>
>
> DIGY
>
> > Race condition in DocumentsWriter.UpdateDocument
> > ------------------------------------------------
> >
> > Key: LUCENENET-135
> > URL: https://issues.apache.org/jira/browse/LUCENENET-135
> > Project: Lucene.Net
> > Issue Type: Bug
> > Environment: 2.3.1
> > Reporter: Digy
> > Attachments: DocumentsWriter+SupportClass.rar,
> DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch,
> SupportClass.patch, SupportClass.patch
> >
> >
> > There is a race condition in DocumentsWriter.UpdateDocument. Locking the
> whole method solves the problem but this is not what is intended in java
> version
> > (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> > I am working on it.
> > DIGY
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615044#action_12615044 ]
Digy commented on LUCENENET-135:
--------------------------------
ok. I got it.
The problem is that java's synchronized methods are converted as
{code}
void method()
{
lock(this)
{
....
}
}
{code}
which is internally implemented as(Monitor.Enter & Monitor.Exit) and java's wait & notify methods are translated as Monitor.Wait & Monitor.PulseAll.
So a method in lock-block and calling Monitor.Wait sends continue-signal to all methods waiting for lock.
I think, patch will not be small.
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: DocumentsWriter+SupportClass.rar
Since calling Monitor.PulseAll or Monitor.Wait before Monitor.Enter results in SynchronizationLockException (which is the case in code after removing "lock(this)"es ) I chose AutoResetEvent that is most suitable for wait/notify.
I also coded a new class "SyncObject" for java's synchronized methods that allows nested locks of the same object in the same thread just like
{code}
lock(this)
{
.....
lock(this)
{
.....
}
}
{code}
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: DocumentsWriter-Temp.patch
DocumentsWriter passes itself to ThreadState while creating, and ThreadState objects access the fields and methods of the DocumentsWriter without locking.
I think, this will not be solved with little changes/corrections.
I'll send a temporary patch.
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-135.
--------------------------
SupportClass.patch applied. No need for other patches.
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Assignee: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615111#action_12615111 ]
Digy commented on LUCENENET-135:
--------------------------------
Forget all about this issue.
The first patch (DocumentsWriter-Temp.patch) seems correct and run concurrently.
DIGY.
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "George Aroush (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618873#action_12618873 ]
George Aroush commented on LUCENENET-135:
-----------------------------------------
Hi DIGY,
The first patch, DocumentsWriter-Temp.patch, diverges from the Java code. Are you this patch is not working around the real issue?
Thanks.
-- George
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: DocumentsWriter.patch
I will attach a new DocumentsWriter.patch, which was applied to lucene.java at March, 3rd 2008 with a comment
"LUCENE-1198: don't increment numDocsInRAM unless DocumentsWriter.ThreadState.init succeeds".
(after the release of 2.3.1).
It is not directly releated with this issue but can be used.
DIGY.
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615034#action_12615034 ]
Digy commented on LUCENENET-135:
--------------------------------
Same bug also exists in ConcurrentMergeScheduler(Sync) which causes TestAddIndexOnDiskFull
{code}
public virtual void Sync()
{
lock (this) //<-------
{
while (MergeThreadCount() > 0)
{
Message("now wait for threads; currently " + mergeThreads.Count + " still running");
int count = mergeThreads.Count;
for (int i = 0; i < count; i++)
Message(" " + i + ": " + ((MergeThread) mergeThreads[i]));
try
{
System.Threading.Monitor.Wait(this); //<-------
}
catch (System.Threading.ThreadInterruptedException)
{
}
}
}
}
{code}
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: SupportClass.patch
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615084#action_12615084 ]
Digy commented on LUCENENET-135:
--------------------------------
Scandal. DEADLOCK!!!
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614980#action_12614980 ]
Digy commented on LUCENENET-135:
--------------------------------
I haven't found a nice solution yet. This example below shows the bug.
DIGY
{code}
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
namespace ConsoleApplication1
{
class MonitorTest
{
static void Main(string[] args)
{
MonitorTest m = new MonitorTest();
m.Start();
}
void Start()
{
Thread t1 = new Thread(new ThreadStart(GetThreadState));
Thread t2 = new Thread(new ThreadStart(GetThreadState));
Thread t3 = new Thread(new ThreadStart(FinishDocument));
t1.IsBackground = t2.IsBackground = t3.IsBackground = true;
t1.Start();
t2.Start();
Thread.Sleep(1000);
t3.Start();
Console.WriteLine("Press ENTER to Exit");
Console.ReadLine();
}
int ThreadCountInCriticalSection = 0;
void GetThreadState()
{
lock (this)
{
ThreadCountInCriticalSection++;
if (ThreadCountInCriticalSection > 1)
Console.WriteLine(ThreadCountInCriticalSection + " threads in critical section");
Monitor.Wait(this); //Unblocks lock(this)
Thread.Sleep(10);
ThreadCountInCriticalSection--;
Console.WriteLine("Leaving Critical section");
}
}
void FinishDocument()
{
lock (this)
{
//Since This thread starts 1 sec later, it should never reach to this point
ThreadCountInCriticalSection++;
if (ThreadCountInCriticalSection > 1)
Console.WriteLine(ThreadCountInCriticalSection + " threads in critical section");
Monitor.PulseAll(this); //Unblocks Monitor.Wait
}
}
}
}
{code}
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620982#action_12620982 ]
Digy commented on LUCENENET-135:
--------------------------------
Hi,
1) SupportClass.patch fixes "Index/TestIndexWriter/TestDocumentsWriterExceptionThreads"
2) In fact , There is no race condition in DocumentsWriter(at least for now). It all started when I saw two threads entering the "GetThreadState" at the same time which is a synchronized method. So, I falsely determined that it was a race condition. And finally, It turned out to be a bug in ThreadClass.
ThreadClass'es static "Current" method always returns a new instance of ThreadClass,
and because it is used as a key to HashTable "threadBindings"(DocumentsWriter.GetThreadState),
{code}
ThreadState state = (ThreadState) threadBindings[SupportClass.ThreadClass.Current()];
if (state == null)
{
...
threadBindings[SupportClass.ThreadClass.Current()] = state;
}
{code}
"State==null" never gets false and ,everytime, initializations of the first-time call are made.
I think the patch is correct. But after applying that patch the following simple test case fails with an exception "Index was outside the bounds of the array".
{code}
[Test]
public void TestIndexWriter1()
{
RAMDirectory dir = new RAMDirectory();
IndexWriter wr = new IndexWriter(dir,new WhitespaceAnalyzer(), true);
wr.SetRAMBufferSizeMB(0.1);
Document doc = new Document();
Field f1 = new Field("field1", "", Field.Store.YES, Field.Index.TOKENIZED);
doc.Add(f1);
for (int i = 0; i < 200000; i++)
{
f1.SetValue("some text, some more text,some text, some more text,some text, some more text,");
try
{
wr.AddDocument(doc);
//if (i % 100 == 0) wr.Flush(); //Make that pass
}
catch (Exception ex)
{
Console.WriteLine("loop:" + i + "\n" + ex.Message + "\n" + ex.StackTrace);
throw ex;
}
}
wr.Close();
}
{code}
*Any Idea?*
PS1:*Uncommenting Flush statement in the code makes that test pass*
PS2: If you decide to close this issue, please take a look at the DocumentsWriter.patch(which has nothing to do with any test case but applied to Lucene.java after the release of 2.3.1)
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615050#action_12615050 ]
Digy commented on LUCENENET-135:
--------------------------------
Patch is too big but it is simple
it changes
{code}
lock(this){
...
}
{code}
to
{code}
Wait();
try
{
....
}
finally{
Release();
}
{code}
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619083#action_12619083 ]
Digy commented on LUCENENET-135:
--------------------------------
A better summary for this issue would be: Bug in static method ThreadClass.Current in SupportClass.cs
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "George Aroush (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620820#action_12620820 ]
George Aroush commented on LUCENENET-135:
-----------------------------------------
Hi DIGY,
I like to know a bit more about this patch:
1) what test case is it fixing?
2) how did you determine the race condition?
The changes to the ThreadClass is considerable and I'm trying to understand why this wasn't a problem in earlier versions.
Thanks.
-- George
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: SupportClass.patch
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619081#action_12619081 ]
Digy commented on LUCENENET-135:
--------------------------------
ThreadClass' static method Current() always returns a new instance of ThreadClass where it should return an instance created only in the current thread.
PS: It is used as a key for Hashtable "threadBindings" in DocumentsWriter.
Patch coming.
DIGY.
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-135:
---------------------------
Attachment: DocumentsWriter.patch
ok. Final(i hope) and simplest patch. No need to change "lock(this)"es and support class.
DIGY
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-135) Race condition in
DocumentsWriter.UpdateDocument
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619088#action_12619088 ]
Digy commented on LUCENENET-135:
--------------------------------
No change in "UpdateDocument"(last DocumentsWriter patch) . TortoiseSVN creates diff-lines for it somehow.
DIGY.
> Race condition in DocumentsWriter.UpdateDocument
> ------------------------------------------------
>
> Key: LUCENENET-135
> URL: https://issues.apache.org/jira/browse/LUCENENET-135
> Project: Lucene.Net
> Issue Type: Bug
> Environment: 2.3.1
> Reporter: Digy
> Attachments: DocumentsWriter+SupportClass.rar, DocumentsWriter-Temp.patch, DocumentsWriter.patch, DocumentsWriter.patch, SupportClass.patch
>
>
> There is a race condition in DocumentsWriter.UpdateDocument. Locking the whole method solves the problem but this is not what is intended in java version
> (testcase: Index/TestIndexWriter/TestDocumentsWriterExceptionThreads)
> I am working on it.
> DIGY
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.