You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Thomas Koch <ko...@orbiteam.de> on 2011/04/14 10:13:01 UTC
failure of some PyLucene tests on windows OS
Hi,
I've observed some tests failing on my windows build of PyLucene.
Tested with PyLucene 2.9(.2) and PyLucene 3.1(.0) (latest) with Python 2.6.
The problem is with test_PyLucene.py and test_PythonDirectory.py and looks
like some file lock issue.
The test fixture tries to cleanup the created index files (tearDown) which
fails with windows complaining the file being still in use by some processes
- see output below (sorry, german windows ,-)
I tried several things like different windows boxes (Vista/Win7), disabling
windows processes (antivir, indexer etc.) - with no luck. Finally I came
across a small change in the test_PyLucene.py code which fixes the issue for
this test:
260c260,261
< self.closeStore(store, reader)
---
> pass
> #self.closeStore(store, reader)
So this looks like a problem in the test-code rather than in
windows/python/pyLucene - some store not being closed results in file lock.
However I couldn't fix the issue in test_PythonDirectory.py and still wonder
if no one else has seen this problem before...
Also I wonder why the tests don't fail on other OSes. So I still fear it
could be somewhat related to either my windows built or windows in
general...
Any ideas welcome... (I guess someone had good reason to comment the
closeStore() ?)
(sample output of failing tests below - I can report the full failure if
that's of any help)
regards,
Thomas
--
OrbiTeam Software GmbH & Co. KG
Endenicher Allee 35
53121 Bonn Germany
http://www.orbiteam.de
sample output:
--------------
test>python test_PyLucene.py
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
======================================================================
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_PyLucene.py", line 304, in tearDown
shutil.rmtree(path) # self.STORE_DIR)
File "C:\Python26\lib\shutil.py", line 221, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "C:\Python26\lib\shutil.py", line 219, in rmtree
os.remove(fullname)
WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
sie von einem anderen Prozess verwendet wird: 'testrepo\\_0.fdt'
==> test succeeds with above fix!
======================================================================
test>python test_PythonDirectory.py
Testing Indexing Incremental Looping
...
indexing 98
indexing 99
.....EEEEEEEE
======================================================================
ERROR: test_removeDocument (__main__.PythonDirectoryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "I:\Devel\test\PyLucene\PYL31\test\test_PyLucene.py", line 178, in
test_removeDocument
self.closeStore(store, searcher, reader)
File "test_PythonDirectory.py", line 241, in closeStore
arg.close()
JavaError: java.lang.RuntimeException: WindowsError
Java stacktrace:
java.lang.RuntimeException: WindowsError
at org.apache.pylucene.store.PythonDirectory.deleteFile(Native
Method)
at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.
java:542)
at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java
:487)
at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java
:475)
at
org.apache.lucene.index.IndexFileDeleter.close(IndexFileDeleter.java:
355)
at
org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java
:838)
at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1191)
at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1177)
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:241)
at org.apache.lucene.index.IndexReader.close(IndexReader.java:1207)
======================================================================
ERROR: test_removeDocument (__main__.PythonDirectoryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_PythonDirectory.py", line 233, in tearDown
shutil.rmtree(self.STORE_DIR)
File "C:\Devel\Python26\lib\shutil.py", line 217, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "C:\Devel\Python26\lib\shutil.py", line 215, in rmtree
os.remove(fullname)
WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
sie
von einem anderen Prozess verwendet wird: 'testpyrepo\\_0.fdt'
======================================================================
RE: failure of some PyLucene tests on windows OS
Posted by Andi Vajda <va...@apache.org>.
On Fri, 15 Apr 2011, Thomas Koch wrote:
>> I'd expect anyone running on Windows to see these test failures.
>>
>> Andi..
>>
> So what do you think about this issue - can we ignore this or claim it's a
> windows bug or hope that 'just' the test code is wrong?
Here is what I think: I think that there is a bug. I don't know where but it
happens only on Windows. I tried to fix it in the past but I don't run
Windows much so I didn't try that hard. For a more definitive answer or
explanation, someone with a stronger itch on Windows is going to have to
step in. Until then, and it's been years, that bug shall remain a bit of a
mystery. Given that no one has reported a more general bug in the same area
of PyLucene indexing on Windows, it could mean that either there are no
users (unlikely) or that this bug is just with the unit test (more likely).
> I'd suggest to at least apply the mentioned fix (i.e. uncomment the close
> call in test_PyLucene) to make this test run on windows. Of course someone
> should confirm this doesn't break the tests on Linux (or other OS)... just
> my thoughts.
I uncommented line 260 in test_PyLucene.py as you suggested and tests seem
to still pass on Mac OS X. I didn't try Linux or Solaris. In fact, I think
that this commented line very much looks like an earlier attempt of mine at
elucidating this bug that I didn't clean up properly.
Checked into rev 1092774 of branch_3x.
Andi..
RE: failure of some PyLucene tests on windows OS
Posted by Thomas Koch <ko...@orbiteam.de>.
> I'd expect anyone running on Windows to see these test failures.
>
> Andi..
>
So what do you think about this issue - can we ignore this or claim it's a
windows bug or hope that 'just' the test code is wrong?
I'd suggest to at least apply the mentioned fix (i.e. uncomment the close
call in test_PyLucene) to make this test run on windows. Of course someone
should confirm this doesn't break the tests on Linux (or other OS)... just
my thoughts.
Regards,
Thomas
Re: failure of some PyLucene tests on windows OS
Posted by Andi Vajda <va...@apache.org>.
On Apr 14, 2011, at 2:22, "Thomas Koch" <ko...@orbiteam.de> wrote:
>> Well, sure, not running the code that breaks solves the problem. But can
> you
>> then run the tests multiple times ?
>>
> [Thomas Koch] note that previously closeStore() was not called, but now when
> calling it the test_PyLucene runs OK.
>
> And yes I can run tests several times- on PyLucene2.9 the index-dirs
> testrepo and testpyrepo are cleaned up (i.e. removed) after tests succeed.
> With PyLucene3.1 the testpyrepo is left (because test_PythonDirectory.py
> fails to cleanup).
>
>>> So this looks like a problem in the test-code rather than in
>>> windows/python/pyLucene - some store not being closed results in file
> lock.
>>
>> Not sure if the bug is there.
>
> Me neither ,-(
>
> Just for test_PyLucene the 'test-code-fix' fixes the issue. Another problem
> is the dependency between test_PythonDirectory and test_PyLucene:
> test_PythonDirectory uses tests defined in test_PyLucene which makes it bit
> difficult to figure out where the problem is...
>
>
> So has anyone seen this problem before?
I'd expect anyone running on Windows to see these test failures.
Andi..
>
>
> Regards,
> Thomas
>
RE: failure of some PyLucene tests on windows OS
Posted by Thomas Koch <ko...@orbiteam.de>.
> Well, sure, not running the code that breaks solves the problem. But can
you
> then run the tests multiple times ?
>
[Thomas Koch] note that previously closeStore() was not called, but now when
calling it the test_PyLucene runs OK.
And yes I can run tests several times- on PyLucene2.9 the index-dirs
testrepo and testpyrepo are cleaned up (i.e. removed) after tests succeed.
With PyLucene3.1 the testpyrepo is left (because test_PythonDirectory.py
fails to cleanup).
> > So this looks like a problem in the test-code rather than in
> > windows/python/pyLucene - some store not being closed results in file
lock.
>
> Not sure if the bug is there.
Me neither ,-(
Just for test_PyLucene the 'test-code-fix' fixes the issue. Another problem
is the dependency between test_PythonDirectory and test_PyLucene:
test_PythonDirectory uses tests defined in test_PyLucene which makes it bit
difficult to figure out where the problem is...
So has anyone seen this problem before?
Regards,
Thomas
Re: failure of some PyLucene tests on windows OS
Posted by Andi Vajda <va...@apache.org>.
On Apr 14, 2011, at 1:13, "Thomas Koch" <ko...@orbiteam.de> wrote:
> Hi,
> I've observed some tests failing on my windows build of PyLucene.
> Tested with PyLucene 2.9(.2) and PyLucene 3.1(.0) (latest) with Python 2.6.
>
> The problem is with test_PyLucene.py and test_PythonDirectory.py and looks
> like some file lock issue.
Yes, I've been aware of this problem for quite some time. It happens on Windows only and I've not tried too hard to fix it hoping that a Windows user might elucidate it.
> The test fixture tries to cleanup the created index files (tearDown) which
> fails with windows complaining the file being still in use by some processes
> - see output below (sorry, german windows ,-)
>
> I tried several things like different windows boxes (Vista/Win7), disabling
> windows processes (antivir, indexer etc.) - with no luck. Finally I came
> across a small change in the test_PyLucene.py code which fixes the issue for
> this test:
>
> 260c260,261
> < self.closeStore(store, reader)
> ---
>> pass
>> #self.closeStore(store, reader)
>
Well, sure, not running the code that breaks solves the problem. But can you then run the tests multiple times ?
> So this looks like a problem in the test-code rather than in
> windows/python/pyLucene - some store not being closed results in file lock.
Not sure if the bug is there.
Andi..
>
> However I couldn't fix the issue in test_PythonDirectory.py and still wonder
> if no one else has seen this problem before...
> Also I wonder why the tests don't fail on other OSes. So I still fear it
> could be somewhat related to either my windows built or windows in
> general...
>
> Any ideas welcome... (I guess someone had good reason to comment the
> closeStore() ?)
>
> (sample output of failing tests below - I can report the full failure if
> that's of any help)
>
> regards,
>
> Thomas
> --
> OrbiTeam Software GmbH & Co. KG
> Endenicher Allee 35
> 53121 Bonn Germany
> http://www.orbiteam.de
>
>
> sample output:
> --------------
>
>
> test>python test_PyLucene.py
>
> EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
> ======================================================================
> ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "test/test_PyLucene.py", line 304, in tearDown
> shutil.rmtree(path) # self.STORE_DIR)
> File "C:\Python26\lib\shutil.py", line 221, in rmtree
> onerror(os.remove, fullname, sys.exc_info())
> File "C:\Python26\lib\shutil.py", line 219, in rmtree
> os.remove(fullname)
> WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
> sie von einem anderen Prozess verwendet wird: 'testrepo\\_0.fdt'
>
> ==> test succeeds with above fix!
>
> ======================================================================
>
> test>python test_PythonDirectory.py
> Testing Indexing Incremental Looping
> ...
> indexing 98
> indexing 99
> .....EEEEEEEE
> ======================================================================
> ERROR: test_removeDocument (__main__.PythonDirectoryTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "I:\Devel\test\PyLucene\PYL31\test\test_PyLucene.py", line 178, in
> test_removeDocument
> self.closeStore(store, searcher, reader)
> File "test_PythonDirectory.py", line 241, in closeStore
> arg.close()
> JavaError: java.lang.RuntimeException: WindowsError
> Java stacktrace:
> java.lang.RuntimeException: WindowsError
> at org.apache.pylucene.store.PythonDirectory.deleteFile(Native
> Method)
> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.
> java:542)
> at
> org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java
> :487)
> at
> org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java
> :475)
> at
> org.apache.lucene.index.IndexFileDeleter.close(IndexFileDeleter.java:
> 355)
> at
> org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java
> :838)
> at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1191)
> at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1177)
> at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:241)
> at org.apache.lucene.index.IndexReader.close(IndexReader.java:1207)
>
>
> ======================================================================
> ERROR: test_removeDocument (__main__.PythonDirectoryTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "test_PythonDirectory.py", line 233, in tearDown
> shutil.rmtree(self.STORE_DIR)
> File "C:\Devel\Python26\lib\shutil.py", line 217, in rmtree
> onerror(os.remove, fullname, sys.exc_info())
> File "C:\Devel\Python26\lib\shutil.py", line 215, in rmtree
> os.remove(fullname)
> WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
> sie
> von einem anderen Prozess verwendet wird: 'testpyrepo\\_0.fdt'
>
> ======================================================================
>
>
>
RE: failure of some PyLucene tests on windows OS
Posted by Thomas Koch <ko...@orbiteam.de>.
I have to add some detail: after applying the mentioned fix to
test_PythonDirectory.py both tests run OK with PyLucene2.9! But with
PyLucene3.1 it's a different story: test_PythonDirectory.py is OK but
test_PythonDirectory.py still fails (see output I sent in previous mail).
Regards,
Thomas
> -----Original Message-----
> From: Thomas Koch [mailto:koch@orbiteam.de]
> Sent: Thursday, April 14, 2011 10:13 AM
> To: pylucene-dev@lucene.apache.org
> Subject: failure of some PyLucene tests on windows OS
>
> Hi,
> I've observed some tests failing on my windows build of PyLucene.
> Tested with PyLucene 2.9(.2) and PyLucene 3.1(.0) (latest) with Python
2.6.
> ...