You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Thomas Koch <ko...@orbiteam.de> on 2012/05/08 17:17:14 UTC
[Why] some PyLucene tests fail on Windows
There's a known issue with some PyLucene tests that fail on Windows -
reported/discussed before - see
http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201104.mbox/<00
0d01cbfa7b$c60ca530$5225ef90$@de>
While some tests have been fixed, some others still show errors / fail on
windows, currently that's
ERROR: test_removeDocument (__main__.PythonDirectoryTests)
FAIL: testTiming
(lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT
est)
I had a look at this and tried to figure out why the PythonDirectoryTests
fail. A possible cause is a windows issue with os.unlink where in certain
situations (timing issue!) a file that is going to be deleted is still
locked by windows - which may happen because of some process being notified
about file removal still holds a lock on it (sth. like an indexer or virus
checker). Another reason could simply be that some tests keep files open...
anyway: "On Windows, attempting to remove a file that is in use causes an
exception to be raised." (from the Python doc on os.remove(path) and
unlink(path)).
The traceback below shows that the test in PythonDirectoryTests fails in
deleteFile (where lock is on an index file like '_0.tis' or '_0.fdt').
There's been a similar problem with Python unit tests (on windows) as
discussed here: http://bugs.python.org/issue7443 (can't see that they came
to a solution though ... it's still open)
I've added special hack for windows to the test_PythonDirectory and added a
try-delete-wait-retry-loop there - this shows that it's probably not a
timing issue (after 10 secs/retries, the file still cannot be deleted) - so
it rather looks like the index files are still open! (couldn't find a bug in
the python code though)
Thus I tried to simply ignore the exception - the files are then *not*
removed, but the test succeeds:
...
indexing 98
indexing 99
failed to delete: testpyrepo\_0.tis
failed to delete: testpyrepo\_0.nrm
failed to delete: testpyrepo\_0.frq
failed to delete: testpyrepo\_0.fdx
failed to delete: testpyrepo\_0.prx
failed to delete: testpyrepo\_0.fdt
failed to delete: testpyrepo\_0.tis
failed to delete: testpyrepo\_0.nrm
failed to delete: testpyrepo\_0.frq
failed to delete: testpyrepo\_0.fdx
failed to delete: testpyrepo\_0.prx
failed to delete: testpyrepo\_0.fdt
...
----------------------------------------------------
Ran 10 tests in 3.659s
(The "failed to delete:" is a debug statement - to be removed)
I could provide a "patch" that allows the test_PythonDirectory to pass on
windows - if that's an accepted solution. Still I see possibility that the
PythonDirectory has a bug (i.e. index files are not closed always) - on the
other hand it could be timing issue or sth related to my environment (?)
I wonder if anyone has used PythonDirectory yet (in production) or if
someone else came across these issues on windows? (I'm not using
PythonDirectory and haven't had similar lock issues except of when running
the tests...) If someone could confirm that the test_PythonDirectory passes
on his/her windows environment that would be good to know ,-)
Haven't had a look at testTiming yet.
Regards,
Thomas
--
Details :
======================================================================
ERROR: test_removeDocument (__main__.PythonDirectoryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"F:\Devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\test\test_PyLucene.
py", line 178, in test_removeDocument
self.closeStore(store, searcher, reader)
File "test_PythonDirectory.py", line 241, in closeStore
arg.close()
JavaError: org.apache.jcc.PythonException: (32, 'Der Prozess kann nicht auf
die Datei zugreifen, da sie von einem anderen Prozess verwendet wird',
u'testpyrepo_0.tis')
Traceback (most recent call last):
File "test_PythonDirectory.py", line 181, in deleteFile
os.unlink(os.path.join(self.path, name))
WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
sie
von einem anderen Prozess verwendet wird: u'testpyrepo\\_0.tis'
Java stacktrace:
...
at org.apache.pylucene.store.PythonDirectory.deleteFile(Native
Method)
at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:57
8)
at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517)
at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:504)
at
org.apache.lucene.index.IndexFileDeleter.close(IndexFileDeleter.java:377)
at
org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:854)
at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1520)
at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1504)
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:221)
at org.apache.lucene.index.IndexReader.close(IndexReader.java:1538)
======================================================================
ERROR: test_removeDocument (__main__.PythonDirectoryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_PythonDirectory.py", line 233, in tearDown
shutil.rmtree(self.STORE_DIR)
File "C:\Python27\lib\shutil.py", line 249, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "C:\Python27\lib\shutil.py", line 247, in rmtree
os.remove(fullname)
WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
sie
von einem anderen Prozess verwendet wird: 'testpyrepo\\_0.fdt'
======================================================================
...
/cygdrive/c/Python27/python.exe
samples/LuceneInAction/CompoundVersusMultiFileIndexTest.py
F
======================================================================
FAIL: testTiming
(lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT
est)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"F:\devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\samples\LuceneInAct
ion\lia\indexing\CompoundVersusMultiFileIndexTest.py", line 62, in
testTiming
self.assert_(cTiming > mTiming)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 3.688s
FAILED (failures=1)
Re: [Why] some PyLucene tests fail on Windows
Posted by Andi Vajda <va...@apache.org>.
On Tue, 8 May 2012, Thomas Koch wrote:
> There's a known issue with some PyLucene tests that fail on Windows -
> reported/discussed before - see
> http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201104.mbox/<00
> 0d01cbfa7b$c60ca530$5225ef90$@de>
>
> While some tests have been fixed, some others still show errors / fail on
> windows, currently that's
>
> ERROR: test_removeDocument (__main__.PythonDirectoryTests)
> FAIL: testTiming
> (lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT
> est)
>
> I had a look at this and tried to figure out why the PythonDirectoryTests
> fail. A possible cause is a windows issue with os.unlink where in certain
> situations (timing issue!) a file that is going to be deleted is still
> locked by windows - which may happen because of some process being notified
> about file removal still holds a lock on it (sth. like an indexer or virus
> checker). Another reason could simply be that some tests keep files open...
> anyway: "On Windows, attempting to remove a file that is in use causes an
> exception to be raised." (from the Python doc on os.remove(path) and
> unlink(path)).
>
> The traceback below shows that the test in PythonDirectoryTests fails in
> deleteFile (where lock is on an index file like '_0.tis' or '_0.fdt').
> There's been a similar problem with Python unit tests (on windows) as
> discussed here: http://bugs.python.org/issue7443 (can't see that they came
> to a solution though ... it's still open)
>
> I've added special hack for windows to the test_PythonDirectory and added a
> try-delete-wait-retry-loop there - this shows that it's probably not a
> timing issue (after 10 secs/retries, the file still cannot be deleted) - so
> it rather looks like the index files are still open! (couldn't find a bug in
> the python code though)
>
> Thus I tried to simply ignore the exception - the files are then *not*
> removed, but the test succeeds:
> ...
> indexing 98
> indexing 99
> failed to delete: testpyrepo\_0.tis
> failed to delete: testpyrepo\_0.nrm
> failed to delete: testpyrepo\_0.frq
> failed to delete: testpyrepo\_0.fdx
> failed to delete: testpyrepo\_0.prx
> failed to delete: testpyrepo\_0.fdt
> failed to delete: testpyrepo\_0.tis
> failed to delete: testpyrepo\_0.nrm
> failed to delete: testpyrepo\_0.frq
> failed to delete: testpyrepo\_0.fdx
> failed to delete: testpyrepo\_0.prx
> failed to delete: testpyrepo\_0.fdt
> ...
> ----------------------------------------------------
> Ran 10 tests in 3.659s
>
> (The "failed to delete:" is a debug statement - to be removed)
>
> I could provide a "patch" that allows the test_PythonDirectory to pass on
> windows - if that's an accepted solution. Still I see possibility that the
> PythonDirectory has a bug (i.e. index files are not closed always) - on the
> other hand it could be timing issue or sth related to my environment (?)
I don't think that hiding failures "under the rug" is the right thing to do.
> I wonder if anyone has used PythonDirectory yet (in production) or if
> someone else came across these issues on windows? (I'm not using
> PythonDirectory and haven't had similar lock issues except of when running
> the tests...) If someone could confirm that the test_PythonDirectory passes
> on his/her windows environment that would be good to know ,-)
PythonDirectory is meant as an example of "python extension" and tests that
the extension is functional. It's not intended as a production-level
implementation.
Andi..
>
>
> Haven't had a look at testTiming yet.
>
> Regards,
> Thomas
> --
> Details :
>
> ======================================================================
> ERROR: test_removeDocument (__main__.PythonDirectoryTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "F:\Devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\test\test_PyLucene.
> py", line 178, in test_removeDocument
> self.closeStore(store, searcher, reader)
> File "test_PythonDirectory.py", line 241, in closeStore
> arg.close()
> JavaError: org.apache.jcc.PythonException: (32, 'Der Prozess kann nicht auf
> die Datei zugreifen, da sie von einem anderen Prozess verwendet wird',
> u'testpyrepo_0.tis')
> Traceback (most recent call last):
> File "test_PythonDirectory.py", line 181, in deleteFile
> os.unlink(os.path.join(self.path, name))
> WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
> sie
> von einem anderen Prozess verwendet wird: u'testpyrepo\\_0.tis'
>
> Java stacktrace:
> ...
> at org.apache.pylucene.store.PythonDirectory.deleteFile(Native
> Method)
> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:57
> 8)
> at
> org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517)
> at
> org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:504)
> at
> org.apache.lucene.index.IndexFileDeleter.close(IndexFileDeleter.java:377)
> at
> org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:854)
> at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1520)
> at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1504)
> at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:221)
> at org.apache.lucene.index.IndexReader.close(IndexReader.java:1538)
>
> ======================================================================
> ERROR: test_removeDocument (__main__.PythonDirectoryTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "test_PythonDirectory.py", line 233, in tearDown
> shutil.rmtree(self.STORE_DIR)
> File "C:\Python27\lib\shutil.py", line 249, in rmtree
> onerror(os.remove, fullname, sys.exc_info())
> File "C:\Python27\lib\shutil.py", line 247, in rmtree
> os.remove(fullname)
> WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
> sie
> von einem anderen Prozess verwendet wird: 'testpyrepo\\_0.fdt'
>
> ======================================================================
>
> ...
>
>
> /cygdrive/c/Python27/python.exe
> samples/LuceneInAction/CompoundVersusMultiFileIndexTest.py
> F
> ======================================================================
> FAIL: testTiming
> (lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT
> est)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "F:\devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\samples\LuceneInAct
> ion\lia\indexing\CompoundVersusMultiFileIndexTest.py", line 62, in
> testTiming
> self.assert_(cTiming > mTiming)
> AssertionError: False is not true
>
> ----------------------------------------------------------------------
> Ran 1 test in 3.688s
>
> FAILED (failures=1)
>
>