Background Index Population Problems on Windows

[Terry Rosenbaum <Te...@amicas.com>]

Hi,

(Using a recent version of Xindice embedded)...

There is a problem with the design for background
index population under windows. The design probably works
OK under UNIX.

The problem is that the thread doing the indexing can
interfere with dropping the indexer or the collection by
holding open file handles while the drop is under way.
The open file handles (index and collection files) prevent
deletion of the file under Windows. The return codes
from the file deletion (in Paged) are never checked so
no errors are reported. But, under Windows, rapid-fire
creation of collection and index, followed by dropping the
same collection can result in the collection's files being
not deleted. Under UNIX, deletion of open files is allowed,
and results in "phantom" files (the files still exist for the
open file handles but no longer appear in the file system
and will be deleted when the last open handle is closed).
Windows works differently -- you cannot delete a file
with an open handle.

The reason I've noticed this is that my application's unit-tests
cause rapid-fire creation/deletion of a collection. The behavior
under Windows leads to a starting condition of a non-empty
collection (since the collection file never actually gets deleted).

Also, the recent change made by Fernando to the IndexManager runs the
PopulateIndexersTimerTask immediately (no delay)
which seems to negate the purpose of delayed background
indexing espoused by Tom Bradford (to allow
population of multiple indexes with a single collection
scan -- several consecutive createIndexer calls queue up
for the background indexer and subsequently a bit
later, all the new indexes are populated in a single collection scan
as opposed to a collection scan per new index).

My earlier proposed patch that did not get applied was to
mitigate (but not entirely fix) this problem. The patch I suggested
was to skip indexing of empty collections entirely thus eliminating
the negative interactions between the index-populating thread
and the main thread creating/dropping collections. With that patch,
the problem was at least avoided when adding indexes to
newly-created empty collections.

I think that we need to have some kind of synchronization between
dropping of collections or indexes and the background index populating
thread to solve this issue. The background indexer needs to be told to stop
populating the index, exit, etc. if the collection is being dropped (or one
of its indexes is being dropped). If the sole purpose of background
indexing is to allow for population of multiple indexes in a single 
collection
scan, then perhaps a better solution would be to make index population
synchronous (any create indexer calls must wait for index population
to complete) but also provide an API to allow for creation of multiple
indexes in a single call. The multiple indexer creation call could create
several indexers but still provide the benefits of populating the multiple
indexers in a single collection scan.

Anyone else have any thoughts/suggestions on this topic?

-Terry