You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "paulirwin (via GitHub)" <gi...@apache.org> on 2026/04/30 12:57:27 UTC
[I] Investigate SimpleFSDirectory file share modes on Windows [lucenenet]
paulirwin opened a new issue, #1283:
URL: https://github.com/apache/lucenenet/issues/1283
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Task description
As part of investigating a concern for #1267, it was discovered that SimpleFSDirectory diverges from the other directory implementations with respect to file share mode on Windows, including in upstream Java Lucene 4.8.1.
In Java, the default file share mode is read/write/delete for FileChannel (used by NIOFS and MMap directories), and read/write for RandomAccessFile (used by SimpleFS). We have matched that on our end (except for MMap which is currently incorrect in master, and will be fixed in #1267 to include delete).
(Note that file share modes are ignored on macOS and Linux, so this only affects Windows. All testing of this issue should be done on Windows, although ensuring no regressions on macOS/Linux via unit tests is always worthwhile.)
It's worth investigating whether this is actually correct or not, despite matching upstream. It might be that this is masking bugs for Windows SimpleFS users around file deletions. i.e. it might be that index files are silently failing to delete and being left on disk, consuming unnecessary space. Or it might be causing other delete operations to fail in ways that harm reliability, data integrity, or availability (i.e. leaking handles).
My hunch is that we should use `FileShare.Delete` across all FS implementations, including SimpleFS, and accept a divergence from Java here to fix potential issues (we're already diverged by our implementations being significantly different, i.e. both SimpleFS and NIOFS in lucenenet use the same FileStream class just with different access methods, whereas Java Lucene implementations use completely different classes). But we should confirm this via tests.
And to quote myself from https://github.com/apache/lucenenet/pull/1267#discussion_r3167988216:
> Also a side note on `FileShare.Delete` to alleviate your concerns: it sounds really scary! Why would we want to allow for someone to delete a file underneath us while we have it mapped?! Well, it doesn't actually do that. If another process/handle tries to delete the file while we have it mapped with file share delete enabled, that other process gets success back as if it were deleted, but we can keep reading/writing it. It just marks it for pending deletion. Once the last file handle is closed, then the file will be lazily deleted. And on Windows, new file handles (via CreateFile) created on a file that is pending deletion will fail. So that helps flush out the active handles. So there's no risk in enabling this, and in fact it can likely help reliability. It doesn't affect our mmap handle either way, even if "deleted" while we have it mapped; it only affects other callers. And again, macOS/Linux don't even validate this anyways and just leaves it open to all modes all the time
, and you don't hear about issues with this on other platforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate SimpleFSDirectory file share modes on Windows [lucenenet]
Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on issue #1283:
URL: https://github.com/apache/lucenenet/issues/1283#issuecomment-4357593560
> we're already diverged by our implementations being significantly different, i.e. both SimpleFS and NIOFS in lucenenet use the same `FileStream` class just with different access methods
Actually, on .NET Core the `NIOFSDirectory` uses the [`RandomAccess`](https://learn.microsoft.com/en-us/dotnet/api/system.io.randomaccess?view=net-10.0) class now and ignores the file stream (for reads). It is just routed through the same `StreamExtensions` method to get there. But it uses the file handle, not the stream to do the read. We have kind of a hacky workaround for .NET Framework, but it doesn't seem worth the effort to backport `RandomAccess` when there are other `FSDirectory` implementations that can be used. Maybe we should consider excluding this class from .NET Standard and .NET Framework or at least find a better way to port it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Delete file share mode to SimpleFSDirectory and FSIndexOutput for Windows support [lucenenet]
Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on issue #1283:
URL: https://github.com/apache/lucenenet/issues/1283#issuecomment-4695803738
This issue was originally about investigating whether we should add the Delete file share mode to SimpleFSDirectory, and the verdict is that we should. We also should add it to FSIndexOutput.
(Note: the report below was generated with Claude Code, but confirms things I had already found independently.)
Key points:
1. The Java behavior we'd diverge from is a known legacy defect, not a design choice
RandomAccessFile lacking FILE_SHARE_DELETE isn't a considered decision the JDK made about correct file semantics; it's an accident of the old java.io native layer that they've never been able to retrofit, because the public RandomAccessFile(File, String mode) constructor has no room to add a share flag without breaking source compatibility. The evidence:
- It's an open OpenJDK enhancement request (JDK-6607535, JDK-6357433) asking them to add it.
- When the JDK got a clean-slate API - NIO.2 FileChannel - they deliberately set shareDelete = true. That's the JVM telling you which behavior it considers correct when it isn't constrained by legacy.
So we're not overriding a deliberate Java design. We're matching Java's own preferred direction (the FileChannel path that NIOFS/MMap already use) and declining to inherit the legacy RandomAccessFile baggage - baggage we have zero obligation to carry, since we don't use RandomAccessFile at all (we use FileStream).
2. It converges all platforms onto one behavior - it doesn't create a Windows special-case, it removes one
Lucene was designed against POSIX semantics: on Linux/macOS, unlink of an open file always succeeds: the name disappears immediately, the inode survives until the last descriptor closes. That's the model IndexFileDeleter assumes. FILE_SHARE_DELETE is precisely the Windows mechanism that reproduces those semantics (pending-delete = remove the name now, free the bytes on last-handle-close).
So the framing "diverges from Java on Windows" undersells it: the change makes Windows behave the same as every other platform. You end up exercising fewer divergent code paths, not more. The current state is the real divergence: Windows is the one OS where deletes mysteriously fail.
3. The cost/benefit is asymmetric: real harm today, no demonstrated harm from the fix
Status-quo harm (concrete): on Windows, a delete against a file a reader holds throws a sharing violation -> IndexFileDeleter swallows it and queues to the deletable retry list (IndexFileDeleter.cs:689-708) -> retried only at writer checkpoints. For the canonical long-lived DirectoryReader (any search server), superseded segments accumulate on disk for the reader's lifetime; disk growth that simply doesn't happen on Linux/macOS.
Fix harm: the adversarial pass hunted for it on real Windows 11, net8.0 and net472 with 7000 iterations and found none: the post-delete File.Exists check doesn't false-trip (pending-delete reports Exists==false immediately), no corruption, no torn reads, no AVE.
Real benefit vs. zero measured cost.
4. FILE_SHARE_DELETE is permissive; it can't break a correctly-behaving caller
This is the safety keystone. A share flag only ever grants other handles more freedom; it never removes capability from our own handle. Our reader/writer keeps full access regardless. The only observable change: when someone else asks to delete the file, Windows says "yes, deferred" instead of "no, sharing violation." And the only "someone else" deleting a Lucene index file is Lucene's own IndexFileDeleter, acting on files it has already determined are unreferenced. Granting that is exactly the intent. The open handle keeps reading valid bytes until it closes: no file is ever yanked out from under an active reader.
5. Delete-share is already the house style
We already diverge from Java structurally: both our SimpleFS and NIOFS use the same FileStream class with different access patterns, nothing like Java's RandomAccessFile-vs-FileChannel split. And FILE_SHARE_DELETE is already what we do almost everywhere:
- NIOFSDirectory read — ReadWrite | Delete (always has)
- WindowsFsyncSupport native fsync — opens with FILE_SHARE_DELETE
- Suggest module's ExternalRefSorter — ReadWrite | Delete
SimpleFSDirectory (and the write path) are the outliers. The change removes an internal inconsistency; it doesn't introduce a divergence.
### Conclusion
We should add Delete file share mode to SimpleFSDirectory and FSIndexOutput. I'll submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Delete file share mode to SimpleFSDirectory and FSIndexOutput for Windows support [lucenenet]
Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin closed issue #1283: Add Delete file share mode to SimpleFSDirectory and FSIndexOutput for Windows support
URL: https://github.com/apache/lucenenet/issues/1283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org