You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2022/02/09 09:58:30 UTC
[GitHub] [lucenenet] hamzashoukat94 opened a new issue #616: Cannot access the cfs file.
hamzashoukat94 opened a new issue #616:
URL: https://github.com/apache/lucenenet/issues/616
**The process cannot access the file 'C:\ProgramData\LuceneIndex-\_0.cfs' because it is being used by another process.**
I'm not able to open the file stream on the CFS file, it is giving me access denied [exception](https://stackoverflow.com/questions/71047400/whos-locking-the-cfs-file-in-lucene).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034080011
Yes, I asked this question on stack overflow.
I'm working on Lucene's structure. I need to copy the index files from one path to another using fileStream. Meanwhile, readers and searchers are opening and closing on index files. Readers are taken from the writer or directory in different applications.
Let's say you have an index on path1, and on-demand, you need to transfer a portion of index files to the second path. That's why I need to open the filestream.
> Is this question also you?
>
> https://stackoverflow.com/questions/71047400/whos-locking-the-cfs-file-in-lucene
>
> It might be helpful if you explain a little more about what you are trying to do. We have the `FileShare` flags set up in a way that the tests pass, but the tests may not cover every use case that is allowed in Lucene, so we might need to adjust some of the settings.
>
> Do note that there are several CFS commands in the [lucene-cli tool](https://lucenenet.apache.org/docs/4.8.0-beta00015/cli/index.html), but in general the assumption (by us) is that the commands will be performed while the index is not being used. However, there shouldn't be any reason why we can't add the `FileShare.Read` flag so multiple file streams can at least read the files (on every file except for `write.lock`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1038697979
I debugged my application code before sending it to you people. I found out that IndexWriter releases access on these files but if any reader is opened on that directory. The reader will not allow me to delete the file. Unfortunately, I don't know the way to close the reader on a specific directory from multiReader (Leaves). If there is a quick way through which I can close or dispose of the reader.
Thanks for the support.
@NightOwl888 @rclabo I Appreciate your quick responses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1055432563
Fair point. it's not uncommon for example code to assume it running on a single thread unless the example code is demoing parts of LuceneNET that are specific to a multi-threaded environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034955598
Thanks for the snippet. However, due to the fact that nearly every component in Lucene is pluggable, short snippets are not much use for tracking issues down. Please provide a *complete* example (that runs) as either a standalone console app or a test. This is so we can be sure we are using the same components that you are.
If you prefer, you may provide a working example in a GitHub repository and provide a link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034538056
> Ok I got your point.
>
> But I need to know why I'm getting the access denied exception. If you can identify then it will be good for me. I tried to close all the instance of IndexWriter, IndexReader and IndexSearcher. But still I'm facing this issue.
I see. Then this is likely some kind of bug, but it is hard to pinpoint in such a large application which of the 50+ `FileStream` usages or other file write methods that could be keeping a file handle open without a [minimal reproducable example](https://stackoverflow.com/help/minimal-reproducible-example), ideally provided as a standalone console app or as a test. In other words, code that we can run *without modification* to observe the issue.
Note that we haven't yet reviewed `IDisposable` usage (see #265), so it is possible we have something hanging open, especially due to the fact that Lucene may be relying on the fact that the [Closable interface](https://docs.oracle.com/javase/8/docs/api/java/io/Closeable.html) inherits the [AutoClosable interface](https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html), which can be configured to automatically clean up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1040547581
Unfortunately, I cannot share the code with you guys.
But @NightOwl888 is right about delaying disposing of the file stream. I'm using MMapDirectory, and I have experienced the latency in disposing of the file stream. Unfortunately, In dotnet, there is no way to find out which process our object is accessing the particular file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034127138
I agree with Shad, If you are trying to make a copy of the index while it's being written to then using the Replicator is probably a good way to go. You may find the info in this issue helpful: https://github.com/apache/lucenenet/issues/401#issuecomment-759258652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1054938606
Yes, I got your point. Then I think we should mention it in our LUCENENET documentation or in index page code samples.
As I checked code samples, I noticed that we are not using any ReferenceManager or SearcherManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034873687
```
IndexWriter writer = null;
lock (_lockObject)
{
if (IndexWriters.ContainsKey(bucketId))
{
writer = IndexWriters[bucketId];
if (writer != null)
{
try
{
// writer.DeleteUnusedFiles();
writer.DeleteAll();
writer.Commit();
writer.Dispose();
}
finally
{
if (IndexWriter.IsLocked(writer.Directory))
{
IndexWriter.Unlock(writer.Directory);
_directory.RemoveSubDirectory(bucketId);
}
writer = `null;`
}
}
IndexWriters.Remove(bucketId);
}
}
}
```
I have shared the code for deleting the index from the source directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034120596
If you just need to copy the index while it is being used, one option is to use the [`Lucene.Net.Replicator` module](https://lucenenet.apache.org/docs/4.8.0-beta00015/api/replicator/Lucene.Net.Replicator.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1039242251
@hamzashoukat94
As an experiment, does the file remain locked if you use `NIOFSDirectory`? It is [using `FileShare.Delete`](https://github.com/apache/lucenenet/blob/f4280e61a89e1e55365b46e89b1778e37f6273d1/src/Lucene.Net/Store/NIOFSDirectory.cs#L105) which should allow deletion of the file even if the file handle is still open.
However, as for why the file handle remains open, that is something that will require a working example to determine.
Although, a quick search indicates that it is probably the underlying OS that has control over this and there isn't much that can be done except to retry after a certain interval.
- https://stackoverflow.com/a/30506866
- https://social.msdn.microsoft.com/Forums/office/en-US/3737fef5-b7b7-434f-b7a7-794b3d4aaa07/is-there-a-delay-between-calling-dispose-on-a-filestream-and-the-file-handle-actually-being?forum=netfxbcl
That said, it would be helpful to know whether the OS eventually releases the file handle, or if it holds it open indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1039538930
If you have a `MultiReader` in order for it to close the Indexes (ie their files) that it has open you need to make sure it decrements the reference counts of all it's SubReaders to 0. Once those reference counts go to 0 it will automatically close the underlying files. Now how exactly to do that is a little less clear to me. As Shad pointed out, trying to answer questions in the abstract is challenging with a system the size of LuceneNET. It's much easier to provide help if you provide a toy chunk of runnable code that demonstrates the issue you are having. This allows us to among other things run the code in the debugger and see the underlying types that are in play. Without that, it's pretty hard to provide concrete specifics to solve an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1054141138
I found out the reason behind that I was creating a indexSearcher instance using indexReader, as there is no dispose of method exists in indexSearcher so, underlying it is not decrementing the reference count inside indexReader which is somehow incremented on the creation of indexSearcher.
There are mutiple options to use indexSearcher.
1. ReferenceManager<>
2. SearcherManager
3. SearcherLifeTimeManager (Only supported for DirectoryReader, not supported for multiReader)
What if the user creates an IndexSearcher directly through its constructor.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1051898390
I have attached the image in which you can see that after disposal there is a subReader Exist that is holding the handle of the file in cfsReader.
```
protected internal override void DoClose()
{
lock (this)
{
IOException ioe = null;
foreach (IndexReader r in GetSequentialSubReaders())
{
try
{
if (closeSubReaders)
{
r.Dispose();
}
else
{
r.DecRef();
}
}
catch (IOException e)
{
if (ioe == null)
{
ioe = e;
}
}
}
// throw the first exception
if (ioe != null)
{
throw ioe;
}
}
}
```
This is the implementation of DoClose() method of MultiReader Class.
I have checked why it is holding a handle till now, so I come up to this point where `refCount.DecrementAndGet()` giving 1 instead of 0. So, it is not closing here.
```
public void DecRef()
{
// only check refcount here (don't call ensureOpen()), so we can
// still close the reader if it was made invalid by a child:
if (refCount.Get() <= 0)
{
throw new ObjectDisposedException(this.GetType().GetTypeInfo().FullName, "this IndexReader is closed");
}
int rc = refCount.DecrementAndGet();
if (rc == 0)
{
closed = true;
Exception throwable = null;
try
{
DoClose();
}
catch (Exception th)
{
throwable = th;
}
finally
{
try
{
ReportCloseToParentReaders();
}
finally
{
NotifyReaderClosedListeners(throwable);
}
}
}
else if (rc < 0)
{
throw new InvalidOperationException("too many decRef calls: refCount is " + rc + " after decrement");
}
}
```
If I call the `r.DoClose()` forcefully, it closed the file handle but next time it will not allow me to open a reader from the writer.
Can you please give me a quick response on it,?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1039538930
If you have a `MultiReader` in order for it to close the Indexes (ie their files) that it has open you need to make sure it decrements the reference counts of all it's SubReaders to 0. Once those reference counts go to 0 it will automatically close the underlying files. Now how exactly to do that is a little less clear to me. As Shad pointed out, trying to answer questions in the abstract is challenging with a system the size of LuceneNET. It's much easier to provide help if you provide a toy chunk of runnable code that demonstrates the issue you are having. This allows us to among other things run the code in the debugger and see the underlying types that are in play. Without that, it's pretty hard to provide concrete specifics to solve an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1040547581
Unfortunately, I cannot share the code with you guys.
But @NightOwl888 is right about delaying disposing of the file stream. I'm using MMapDirectory, and I have experienced the latency in disposing of the file stream. Unfortunately, In dotnet, there is no way to find out which process our object is accessing the particular file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1054364945
`SearcherManager` help you manage the use of `IndexSearchers` and their underlying `IndexReader` across threads. When multiple threads hold references to objects that have references to `IndexReaders` the system needs some way to know if the underlying `LeafReaders` can be release or not. This is the purpose of the `ReferenceManager` and the incrementing and decrementing of reference counts. If you don't use the `SearcherManager` and hence the `ReferenceManager` it inherits from then it's up to you to manage the lifetime of the object that you use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1039242251
@hamzashoukat94
As an experiment, does the file remain locked if you use `NIOFSDirectory`? It is [using `FileShare.Delete`](https://github.com/apache/lucenenet/blob/f4280e61a89e1e55365b46e89b1778e37f6273d1/src/Lucene.Net/Store/NIOFSDirectory.cs#L105) which should allow deletion of the file even if the file handle is still open.
However, as for why the file handle remains open, that is something that will require a working example to determine.
Although, a quick search indicates that it is probably the underlying OS that has control over this and there isn't much that can be done except to retry after a certain interval.
- https://stackoverflow.com/a/30506866
- https://social.msdn.microsoft.com/Forums/office/en-US/3737fef5-b7b7-434f-b7a7-794b3d4aaa07/is-there-a-delay-between-calling-dispose-on-a-filestream-and-the-file-handle-actually-being?forum=netfxbcl
That said, it would be helpful to know whether the OS eventually releases the file handle, or if it holds it open indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034480865
Ok I got your point.
But I need to know why I'm getting the access denied exception. If you can identify then it will be good for me. I tried to close all the instance of IndexWriter, IndexReader and IndexSearcher. But still I'm facing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] hamzashoukat94 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
hamzashoukat94 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034887280
Why `indexWriter.Dispose()` does not release the handle from the cfs file.?
Additionally, I'm also unlocking the writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1033913053
I suspect you are trying to open the index via more than one `IndexWriter`. This is not allowed. You can have multiple threads writing to the index but they need to do it through the same `IndexWriter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #616: Cannot access the cfs file.
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #616:
URL: https://github.com/apache/lucenenet/issues/616#issuecomment-1034001416
Is this question also you?
https://stackoverflow.com/questions/71047400/whos-locking-the-cfs-file-in-lucene
It might be helpful if you explain a little more about what you are trying to do. We have the `FileShare` flags set up in a way that the tests pass, but the tests may not cover every use case that is allowed in Lucene, so we might need to adjust some of the settings.
Do note that there are several CFS commands in the [lucene-cli tool](https://lucenenet.apache.org/docs/4.8.0-beta00015/cli/index.html), but in general the assumption (by us) is that the commands will be performed while the index is not being used. However, there shouldn't be any reason why we can't add the `FileShare.Read` flag so multiple file streams can at least read the files (on every file except for `write.lock`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org