You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "paulirwin (via GitHub)" <gi...@apache.org> on 2024/05/11 04:34:50 UTC

[PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

paulirwin opened a new pull request, #938:
URL: https://github.com/apache/lucenenet/pull/938

   <!-- Thank you for submitting a pull request to our repo. -->
   
   <!-- Please do NOT submit PRs for features in newer versions of Lucene. We should keep the target version consistent across our repository to make the upgrade process to newer versions of Lucene go smoothly. -->
   
   <!-- If this is your first PR in the Lucene.NET repo, please run through the checklist
   below to ensure a smooth review and merge process for your PR. -->
   
   - [X] You've read the [Contributor Guide](https://github.com/apache/lucenenet/blob/main/CONTRIBUTING.md) and [Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
   - [X] You've included unit or integration tests for your change, where applicable. (Existing unit tests)
   - [X] You've included inline docs for your change, where applicable.
   - [X] There's an open issue for the PR that you are making. If you'd like to propose a change, please [open an issue](https://github.com/apache/lucenenet/issues/new/choose) to discuss the change or find an existing issue.
   
   <!-- Once all that is done, you're ready to go. Open the PR with the content below. -->
   
   This restores the commented-out fsync behavior in FSDirectory to help mitigate a performance regression in .NET 8.
   
   Fixes #933
   
   ## Description
   
   See 933 for more details. Perf benchmark before:
   ```
   | Method                  | Job      | Runtime  | IterationCount | LaunchCount | WarmupCount | N  | Mean      | Error      | StdDev   | Median    | Ratio | RatioSD |
   |------------------------ |--------- |--------- |--------------- |------------ |------------ |--- |----------:|-----------:|---------:|----------:|------:|--------:|
   | IndexDocumentsBenchmark | .NET 7.0 | .NET 7.0 | Default        | Default     | Default     | 10 |  44.11 ms |   3.756 ms | 11.02 ms |  40.34 ms |  1.00 |    0.00 |
   | IndexDocumentsBenchmark | .NET 8.0 | .NET 8.0 | Default        | Default     | Default     | 10 | 843.36 ms |  16.485 ms | 24.16 ms | 843.93 ms | 19.11 |    4.86 |
   ```
   
   After:
   ```
   | Method                  | Job      | Runtime  | IterationCount | LaunchCount | WarmupCount | N  | Mean     | Error      | StdDev    | Median   | Ratio | RatioSD |
   |------------------------ |--------- |--------- |--------------- |------------ |------------ |--- |---------:|-----------:|----------:|---------:|------:|--------:|
   | IndexDocumentsBenchmark | .NET 7.0 | .NET 7.0 | Default        | Default     | Default     | 10 | 50.15 ms |   5.044 ms | 14.874 ms | 54.12 ms |  1.00 |    0.00 |
   | IndexDocumentsBenchmark | .NET 8.0 | .NET 8.0 | Default        | Default     | Default     | 10 | 42.88 ms |   4.736 ms | 13.964 ms | 38.17 ms |  0.95 |    0.50 |
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713961018


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -88,6 +88,21 @@ public int Count
             }
         }
 
+        private int CountInternal
+        {
+            get
+            {
+                int count = 0;
+
+                foreach (var t in _tables.CountPerLock)

Review Comment:
   See above, marking resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2113908227

   The performance is even better with this new approach. Of course some heavily multi-threaded writes/syncs might see a hit, but at least it's more thread-safe now.
   
   ```
   | Method                  | Job      | Runtime  | IterationCount | LaunchCount | WarmupCount | N  | Mean     | Error     | StdDev   | Ratio | RatioSD |
   |------------------------ |--------- |--------- |--------------- |------------ |------------ |--- |---------:|----------:|---------:|------:|--------:|
   | IndexDocumentsBenchmark | .NET 7.0 | .NET 7.0 | Default        | Default     | Default     | 10 | 32.06 ms |  0.771 ms | 2.135 ms |  1.00 |    0.00 |
   | IndexDocumentsBenchmark | .NET 8.0 | .NET 8.0 | Default        | Default     | Default     | 10 | 31.57 ms |  0.781 ms | 2.189 ms |  0.99 |    0.09 |
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713951955


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -74,9 +74,9 @@ public int Count
                 {
                     AcquireAllLocks(ref acquiredLocks);
 
-                    for (var i = 0; i < _tables.CountPerLock.Length; i++)
+                    foreach (var t in _tables.CountPerLock)

Review Comment:
   ConcurrentDictionary does use foreach: https://github.com/dotnet/runtime/blob/c4624d0a3c91fb1496b0910b22c9d9fc197247e8/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs#L1180-L1189
   
   Given that Microsoft does use foreach in ConcurrentDictionary (now, anyways), I have demonstrated that it is faster, and that this aligns with Microsoft's guidance (while acknowledging that they point out that LINQ Sum can be even faster due to vectorization, not to mention simpler code), can we leave this as a foreach?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597753045


##########
src/Lucene.Net/Support/IO/WindowsFsyncSupport.cs:
##########
@@ -0,0 +1,119 @@
+using System;
+using System.IO;
+using System.Runtime.InteropServices;
+
+namespace Lucene.Net.Support.IO
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    public static class WindowsFsyncSupport

Review Comment:
   I forgot to drop the link: https://github.com/dotnet/runtime/tree/42c2362ca7d6030a4f665bb309fe49ce6552502b/src/libraries/Common/src/Interop. Also forgot to mention that they are using nested *partial* classes. So, all code sees the native methods in the same class, but each method call is in a separate file. Well organized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713887342


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -74,9 +74,9 @@ public int Count
                 {
                     AcquireAllLocks(ref acquiredLocks);
 
-                    for (var i = 0; i < _tables.CountPerLock.Length; i++)
+                    foreach (var t in _tables.CountPerLock)

Review Comment:
   I am not suggesting we spend a lot of time optimizing this. Microsoft has already optimized their collections quite a bit (and this is directly based on ConcurrentDictionary). So, it is usually best to follow their lead. I have had to borrow enough collections to know that they never use foreach. I suspect there are very good reasons for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1714104656


##########
src/Lucene.Net/Support/Native/Interop.Posix.Close.cs:
##########
@@ -0,0 +1,31 @@
+using System.Runtime.InteropServices;
+
+namespace Lucene.Net.Support.Native
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    internal static partial class Interop
+    {
+        internal static partial class Posix

Review Comment:
   Gotcha. Let's just stick with this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713805554


##########
src/Lucene.Net/Support/Native/Interop.Posix.Close.cs:
##########
@@ -0,0 +1,31 @@
+using System.Runtime.InteropServices;
+
+namespace Lucene.Net.Support.Native
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    internal static partial class Interop
+    {
+        internal static partial class Posix

Review Comment:
   These methods that all UNIX-y platforms have in common are defined by the POSIX standard, see the spec URLs I added to i.e. `close`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1600871908


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   Working on finishing the ExceptWith implementation now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713958584


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -74,9 +74,9 @@ public int Count
                 {
                     AcquireAllLocks(ref acquiredLocks);
 
-                    for (var i = 0; i < _tables.CountPerLock.Length; i++)
+                    foreach (var t in _tables.CountPerLock)

Review Comment:
   Gotcha. Sure, leave this as-is. They probably have made optimizations at lower levels since the days they used for loops for everything in collections.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713960304


##########
src/Lucene.Net/Support/Native/Interop.MacOS.Constants.cs:
##########
@@ -0,0 +1,28 @@
+namespace Lucene.Net.Support.Native

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597761803


##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -516,8 +519,53 @@ public static void ReThrowUnchecked(Exception th)
             }
         }
 
-        // LUCENENET specific: Fsync is pointless in .NET, since we are
-        // calling FileStream.Flush(true) before the stream is disposed
-        // which means we never need it at the point in Java where it is called.
+        public static void Fsync(FileSystemInfo fileToSync, bool isDir)
+        {
+            // LUCENENET NOTE: there is a bug in 4.8 where it tries to fsync a directory on Windows,
+            // which is not supported in OpenJDK. This change adopts the latest Lucene code as of 9.10
+            // and only fsyncs directories on Linux and macOS.
+
+            // If the file is a directory we have to open read-only, for regular files we must open r/w for
+            // the fsync to have an effect.
+            // See http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/
+            if (isDir && Constants.WINDOWS)
+            {
+                // opening a directory on Windows fails, directories can not be fsynced there
+                if (System.IO.Directory.Exists(fileToSync.FullName) == false)
+                {
+                    // yet do not suppress trying to fsync directories that do not exist
+                    throw new DirectoryNotFoundException($"Directory does not exist: {fileToSync}");
+                }
+                return;
+            }
+
+            try
+            {
+                // LUCENENET specific: was: file.force(true);
+                // We must call fsync on the parent directory, requiring some custom P/Invoking
+                if (Constants.WINDOWS)
+                {
+                    WindowsFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+                else
+                {
+                    PosixFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+            }
+            catch (Exception e) when (e.IsIOException())
+            {
+                if (isDir)
+                {
+                    Debug.Assert((Constants.LINUX || Constants.MAC_OS_X) == false,

Review Comment:
   Yes. In Java, both JUnit and the `assert` statements throw the same `AssertionError` type. Furthermore, in Java the asserts can be (and are) turned on during testing. We test on Release builds, so there is no way to emulate this with `Debug.Assert`. Some of the asserts in Lucene are *obviously* test conditions because they keep track of state and do asserts only when asserts are turned on. But, for the most part it is impossible to tell whether they are intended:
   
   1. As test conditions
   2. For debugging
   3. As guard clauses that can be "turned off" in production
   
   Or any combination of the above.
   
   Since we cannot tell what the intention is from the code, it is best to always run the asserts during testing so we can be sure we are testing under the same conditions as in Lucene. It also ensures end users can do the same tests as in Lucene by turning on the asserts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597692059


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))
+            {
+                Clear();
+                return;
+            }
+
+            foreach (var item in other)
+            {
+                TryRemove(item);

Review Comment:
   Agreed, I did add the `other == this` (just with a `ReferenceEquals` call instead) optimization. (btw, for posterity, your link goes to `IntersectWith`, but the `ExceptWith` code is right below it.) I didn't add the `Count` optimization because that needs to block the entire set and I wasn't yet prepared to implement a full blocking implementation once I considered the concerns above. And as noted above, it's probably a moot point if we end up removing the use of our `ConcurrentHashSet` because of the race condition issues I noted. I'm now suspicious of other uses of this type btw.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597694400


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   Also by "synchronize around the entire hashset" I mean taking a lock around not just the hashset but the operations that affect the files in the hashset (i.e. DeleteFile, Sync, etc) - which could potentially be expensive in a highly multithreaded app but would ensure that all files are fsynced when Sync is called. The alternative is not caring about some possible fsync misses (which appears to be the case with Java Lucene code, as it would have the same issues).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597830128


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))
+            {
+                Clear();
+                return;
+            }
+
+            foreach (var item in other)
+            {
+                TryRemove(item);

Review Comment:
   > I'm now suspicious of other uses of this type btw.
   
   Well, you should be :blush:. Most of the concurrency problems we found were due to incorrect lock usage, but these issues are difficult to track down. I suspect that #935 is due to a missing lock somewhere, but it could also be some sort of timing issue. There are 3 locking scenarios that are most likely:
   
   - A collection type is not being locked in a similar enough way to Java to meet our requirements. That is, an issue with the design of the collection that was chosen to replace the one in Lucene.
   - A collection is being used as a lock object, but the collection is not implemented in a way where it is sharing the lock internally. To fix these, some collections were set up to use the [`SyncRoot` property](https://learn.microsoft.com/en-us/dotnet/api/system.collections.icollection.syncroot?view=net-8.0), but I am not sure whether that is the best solution.
   - A type was changed from `lock (this)` to `lock (syncLock)` (or in our case `UninterruptableMonitor.Enter(syncLock)`) and the lock is not being shared everywhere it was in Lucene. In particular, we need to look out for inherited types that should be sharing the same lock as its base class and external code that uses the object as a lock object.
   
   Fortunately, #935 is the only known concurrency bug in the core assembly and fixing it is not a breaking change so it is not blocking the release.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597688147


##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -516,8 +519,53 @@ public static void ReThrowUnchecked(Exception th)
             }
         }
 
-        // LUCENENET specific: Fsync is pointless in .NET, since we are
-        // calling FileStream.Flush(true) before the stream is disposed
-        // which means we never need it at the point in Java where it is called.
+        public static void Fsync(FileSystemInfo fileToSync, bool isDir)
+        {
+            // LUCENENET NOTE: there is a bug in 4.8 where it tries to fsync a directory on Windows,
+            // which is not supported in OpenJDK. This change adopts the latest Lucene code as of 9.10
+            // and only fsyncs directories on Linux and macOS.
+
+            // If the file is a directory we have to open read-only, for regular files we must open r/w for
+            // the fsync to have an effect.
+            // See http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/
+            if (isDir && Constants.WINDOWS)
+            {
+                // opening a directory on Windows fails, directories can not be fsynced there
+                if (System.IO.Directory.Exists(fileToSync.FullName) == false)
+                {
+                    // yet do not suppress trying to fsync directories that do not exist
+                    throw new DirectoryNotFoundException($"Directory does not exist: {fileToSync}");
+                }
+                return;
+            }
+
+            try
+            {
+                // LUCENENET specific: was: file.force(true);
+                // We must call fsync on the parent directory, requiring some custom P/Invoking
+                if (Constants.WINDOWS)
+                {
+                    WindowsFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+                else
+                {
+                    PosixFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+            }
+            catch (Exception e) when (e.IsIOException())
+            {
+                if (isDir)
+                {
+                    Debug.Assert((Constants.LINUX || Constants.MAC_OS_X) == false,

Review Comment:
   Are you sure we should use `Debugging`? `Debug.Assert` will not be emitted in Release builds, while AFAICT `Debugging` calls would because they are not `[Conditional("DEBUG")]`. There is no production concern (nor string concatenation concern) with `Debug.Assert`. That said, I'd almost rather remove the line entirely because we already check for Windows now above. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597773408


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   My concern is less about whether this will succeed with FSync, and more about whether the `ConcurrentHashSet<T>` matches the behavior in Java, as it may be eventually cleaned up, moved to J2N, and have a full suite of Harmony + dotnet/runtime tests added to confirm it works as expected.
   
   Synchronizing around the entire set is what is done in Java. They use a wrapper class that locks access to the entire set. But IMO, we are better off emulating the `ConcurrentDictionary` in .NET, which uses locked "buckets". The whole set can be locked by locking all of the buckets.
   
   So, like what was done on `UnionWith`, if we are calling into existing code, we should separate the internal operations from the locking code so we can call into the business logic under different locking conditions. In this case, we should be locking all of the buckets at the beginning of the method call and then calling `InternalRemove` (which does no locking).
   
   > I don't think there's any amount of concurrent-izing this hashset that avoids these potential race conditions. The only proper solution would be to synchronize around the entire hashset, at which point we might as well just use the BCL non-concurrent HashSet instead. Let me know what you think.
   
   I guess that is also an option. The `ConcurrentSet` is just such a wrapper that locks the entire set as was done in Java. It can be used by calling `new HashSet<T>().AsConcurrent()`.
   
   But, it mostly only exists because of other JDK concerns such as structural equality, structural formatting, and because these `ISet<T>` operations are fully implemented.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2111511168

   Looks like it still fails on Windows sometimes. I think it is more robust than it was, though. I had to try with `[Repeat(25)]` to get it to fail. I will try in Azure DevOps with the attribute to see whether it will fail on Linux or macOS.
   
   ```console
    TestNRTThreads_Mem
      Source: TestIndexWriterOnJRECrash.cs line 51
      Duration: 34.3 sec
   
     Message: 
   Lucene.Net.Util.LuceneSystemException : CheckIndex failed
   
   To reproduce this test result:
   
   Option 1:
   
    Apply the following assembly-level attributes:
   
   [assembly: Lucene.Net.Util.RandomSeed("0x58551ebf9b5e10f6")]
   [assembly: NUnit.Framework.SetCulture("en-AG")]
   
   Option 2:
   
    Use the following .runsettings file:
   
   <RunSettings>
     <TestRunParameters>
       <Parameter name="tests:seed" value="0x58551ebf9b5e10f6" />
       <Parameter name="tests:culture" value="en-AG" />
     </TestRunParameters>
   </RunSettings>
   
   See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
   
     Stack Trace: 
   TestUtil.CheckIndex(Directory dir, Boolean crossCheckTermVectors) line 177
   TestUtil.CheckIndex(Directory dir) line 162
   TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 322
   TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 329
   TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 329
   TestIndexWriterOnJRECrash.TestNRTThreads_Mem() line 88
   InvokeStub_TestIndexWriterOnJRECrash.TestNRTThreads_Mem(Object, Object, IntPtr*)
   MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   
     Standard Output: 
   CheckIndex failed
   Segments file=segments_2 numSegments=3 version=4.8 format=
     1 of 3: name=_9 docCount=162
       codec=Lucene46
       compound=True
       numFiles=4
       size (MB)=0.34093189239501953
       diagnostics = {source=merge, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=8.0.1, java.vendor=Microsoft, timestamp=15/05/2024 10:17:05 am, mergeMaxNumSegments=-1, mergeFactor=5}
       has deletions [delGen=1]
       test: open reader.........OK
       test: check integrity.....OK
       test: check live docs.....OK [32 deleted docs]
       test: fields..............OK [26 fields]
       test: field norms.........OK [21 fields]
       test: terms, freq, prox...OK [6532 terms; 9562 terms/docs pairs; 12115 tokens]
       test (ignoring deletes): terms, freq, prox...OK [7799 terms; 11823 terms/docs pairs; 14862 tokens]
       test: stored fields.......OK [660 total field count; avg 5.076923 fields per doc]
       test: term vectors........OK [265 total vector count; avg 2.0384614 term/freq vector fields per doc]
       test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
   
     2 of 3: name=_8 docCount=56
       codec=Lucene46
       compound=False
       numFiles=33
       size (MB)=0.18136310577392578
       diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=8.0.1, java.vendor=Microsoft, timestamp=15/05/2024 10:17:06 am}
       has deletions [delGen=1]
       test: open reader.........OK
       test: check integrity.....OK
       test: check live docs.....OK [2 deleted docs]
       test: fields..............OK [13 fields]
       test: field norms.........OK [8 fields]
       test: terms, freq, prox...OK [3962 terms; 4916 terms/docs pairs; 7363 tokens]
       test (ignoring deletes): terms, freq, prox...OK [4035 terms; 5024 terms/docs pairs; 7480 tokens]
       test: stored fields.......OK [282 total field count; avg 5.2222223 fields per doc]
       test: term vectors........OK [112 total vector count; avg 2.074074 term/freq vector fields per doc]
       test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
   
     3 of 3: name=_7 docCount=44
       codec=Lucene46
       compound=False
       numFiles=31
       size (MB)=0.5951318740844727
       diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=8.0.1, java.vendor=Microsoft, timestamp=15/05/2024 10:17:06 am}
       has deletions [delGen=1]
       test: open reader.........OK
       test: check integrity.....OK
       test: check live docs.....OK [1 deleted docs]
       test: fields..............OK [18 fields]
       test: field norms.........OK [13 fields]
       test: terms, freq, prox...ERROR: Lucene.Net.Diagnostics.AssertionException: buffer.length=190007630 count=2142165037
      at Lucene.Net.Codecs.MockIntBlock.MockVariableInt32BlockPostingsFormat.MockInt32Factory.VariableInt32BlockIndexInputAnonymousClass.BlockReaderAnonymousClass.ReadBlock() in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockIntBlock\MockVariableIntBlockPostingsFormat.cs:line 110
      at Lucene.Net.Codecs.IntBlock.VariableInt32BlockIndexInput.Reader.MaybeSeek() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\VariableIntBlockIndexInput.cs:line 152
      at Lucene.Net.Codecs.IntBlock.VariableInt32BlockIndexInput.Reader.Next() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\VariableIntBlockIndexInput.cs:line 160
      at Lucene.Net.Codecs.Sep.SepPostingsReader.SepDocsAndPositionsEnum.NextDoc() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 561
      at Lucene.Net.Index.CheckIndex.CheckFields(Fields fields, IBits liveDocs, Int32 maxDoc, FieldInfos fieldInfos, Boolean doPrint, Boolean isVectors, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1156
      at Lucene.Net.Index.CheckIndex.TestPostings(AtomicReader reader, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1651
   Lucene.Net.Diagnostics.AssertionException: buffer.length=190007630 count=2142165037
      at Lucene.Net.Codecs.MockIntBlock.MockVariableInt32BlockPostingsFormat.MockInt32Factory.VariableInt32BlockIndexInputAnonymousClass.BlockReaderAnonymousClass.ReadBlock() in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockIntBlock\MockVariableIntBlockPostingsFormat.cs:line 110
      at Lucene.Net.Codecs.IntBlock.VariableInt32BlockIndexInput.Reader.MaybeSeek() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\VariableIntBlockIndexInput.cs:line 152
      at Lucene.Net.Codecs.IntBlock.VariableInt32BlockIndexInput.Reader.Next() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\VariableIntBlockIndexInput.cs:line 160
      at Lucene.Net.Codecs.Sep.SepPostingsReader.SepDocsAndPositionsEnum.NextDoc() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 561
      at Lucene.Net.Index.CheckIndex.CheckFields(Fields fields, IBits liveDocs, Int32 maxDoc, FieldInfos fieldInfos, Boolean doPrint, Boolean isVectors, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1156
      at Lucene.Net.Index.CheckIndex.TestPostings(AtomicReader reader, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1651
       test: stored fields.......OK [221 total field count; avg 5.139535 fields per doc]
       test: term vectors........ERROR [Exception of type 'Lucene.Net.Diagnostics.AssertionException' was thrown.]
   Lucene.Net.Diagnostics.AssertionException: Exception of type 'Lucene.Net.Diagnostics.AssertionException' was thrown.
      at Lucene.Net.Codecs.IntBlock.FixedInt32BlockIndexInput.Index.Read(DataInput indexIn, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\FixedIntBlockIndexInput.cs:line 177
      at Lucene.Net.Codecs.Sep.SepPostingsReader.DecodeTerm(Int64[] empty, DataInput input, FieldInfo fieldInfo, BlockTermState termState, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 202
      at Lucene.Net.Codecs.BlockTerms.BlockTermsReader.FieldReader.SegmentTermsEnum.DecodeMetaData() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\BlockTerms\BlockTermsReader.cs:line 986
      at Lucene.Net.Codecs.BlockTerms.BlockTermsReader.FieldReader.SegmentTermsEnum.DocsAndPositions(IBits liveDocs, DocsAndPositionsEnum reuse, DocsAndPositionsFlags flags) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\BlockTerms\BlockTermsReader.cs:line 798
      at Lucene.Net.Index.TermsEnum.DocsAndPositions(IBits liveDocs, DocsAndPositionsEnum reuse) in F:\Projects\lucenenet\src\Lucene.Net\Index\TermsEnum.cs:line 242
      at Lucene.Net.Index.CheckIndex.TestTermVectors(AtomicReader reader, TextWriter infoStream, Boolean verbose, Boolean crossCheckTermVectors) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 2146
       test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
   FAILED
       WARNING: fixIndex() would remove reference to this segment; full exception:
   Lucene.Net.Util.LuceneSystemException: Term Index test failed
      at Lucene.Net.Index.CheckIndex.DoCheckIndex(IList`1 onlySegments) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 848
   
   WARNING: 1 broken segments (containing 43 documents) detected
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2113900014

   @NightOwl888 I just pushed up a change that synchronizes access to the HashSet, let me know what you think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin merged PR #938:
URL: https://github.com/apache/lucenenet/pull/938


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713134026


##########
src/Lucene.Net.Tests/Index/TestIndexWriterOnJRECrash.cs:
##########
@@ -42,16 +43,16 @@ namespace Lucene.Net.Index
     [TestFixture]
     public class TestIndexWriterOnJRECrash : TestNRTThreads
     {
-        // LUCENENET: Setup unnecessary because we create a new temp directory
+        // LUCENENET: Setup of directory unnecessary because we create a new temp directory
         // in each iteration of the test.
 
         [Test]
         [Slow]
-        [AwaitsFix]
+        [Repeat(25)]

Review Comment:
   Please remove `[Repeat(25)]` and re-add `[AwaitsFix]`, since this is something we will need to revisit in #894.



##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -88,6 +88,21 @@ public int Count
             }
         }
 
+        private int CountInternal
+        {
+            get
+            {
+                int count = 0;
+
+                foreach (var t in _tables.CountPerLock)

Review Comment:
   I didn't benchmark, but it seems that using a for loop would be faster because we don't need to new up an enumerator. Or at least that is what I would do in absence of a benchmark, since that is what the original author and Microsoft did.



##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -74,9 +74,9 @@ public int Count
                 {
                     AcquireAllLocks(ref acquiredLocks);
 
-                    for (var i = 0; i < _tables.CountPerLock.Length; i++)
+                    foreach (var t in _tables.CountPerLock)

Review Comment:
   I didn't benchmark, but it seems that using a for loop would be faster because we don't need to new up an enumerator. Or at least that is what I would do in absence of a benchmark, since that is what the original author and Microsoft did.



##########
src/Lucene.Net.Tests/Support/TestConcurrentHashSet.cs:
##########
@@ -0,0 +1,51 @@
+using Lucene.Net.Attributes;
+using Lucene.Net.Support;
+using NUnit.Framework;
+using System.Linq;
+using System.Threading.Tasks;
+
+namespace Lucene.Net
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    public class TestConcurrentHashSet

Review Comment:
   I added a link in #920 so this can be revisited.



##########
src/Lucene.Net/Support/Native/Interop.MacOS.Constants.cs:
##########
@@ -0,0 +1,28 @@
+namespace Lucene.Net.Support.Native

Review Comment:
   Ideally, there wouldn't be a `Lucene.Net.Support` namespace (there is a Support folder to physically separate our changes from the Java source, but the trend is to put them into more logical namespaces). Let's change these to be
   
   `namespace Lucene.Net.Native`



##########
src/Lucene.Net/Support/Native/Interop.Posix.Close.cs:
##########
@@ -0,0 +1,31 @@
+using System.Runtime.InteropServices;
+
+namespace Lucene.Net.Support.Native
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    internal static partial class Interop
+    {
+        internal static partial class Posix

Review Comment:
   What is the reason for using `Posix` here? Note that Microsoft separates `Linux` and `Unix` into separate classes. I don't know if that means we have to, but I just thought I would check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1713836719


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -74,9 +74,9 @@ public int Count
                 {
                     AcquireAllLocks(ref acquiredLocks);
 
-                    for (var i = 0; i < _tables.CountPerLock.Length; i++)
+                    foreach (var t in _tables.CountPerLock)

Review Comment:
   There was a Stephen Toub video recently where he was showing how you shouldn't prematurely optimize things like this, because the compiler can sometimes come up with something even faster.
   
   I wrote a small benchmark and strictly for summing over an int array, foreach is faster, and LINQ `Sum()` is even faster:
   ```
   |-------- |---------:|---------:|---------:|---------:|
   | ForEach | 68.36 ns | 1.377 ns | 2.750 ns | 67.69 ns |
   | For     | 89.54 ns | 1.820 ns | 3.956 ns | 89.10 ns |
   | LinqSum | 50.46 ns | 1.322 ns | 3.878 ns | 48.67 ns |
   ```
   
   I could do the LINQ Sum here, but let me know if you're good with this for now.
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2113074427

   Well, the good news is that this seems to be 100% stable on .NET Framework.
   
   The bad news is that it is not performing as well on .NET Core. I attempted to swap the `m_staleFiles` set with `new JCG.HashSet<T>().AsConcurrent()` and it seemed to make Windows worse, but Linux and macOS better (NOTE: I rolled that change back). They still all randomly fail, but it is on the order of:
   
   - Windows: Once every ~150 power failures
   - Linux: Once every ~350 power failures
   - macOS: Once every ~500 power failures
   
   This is still a huge improvement over 4.8.0-beta00016, which would get a corrupt index about once every ~30 power failures.
   
   Here are the results of the runs with the swapped hashset implementation with `[Repeat(25)]` (which runs the test 250 times, since the original test runs it 10 times):
   
   https://dev.azure.com/shad0962/Experiments/_build/results?buildId=2343&view=results
   https://dev.azure.com/LuceneNET-Temp/Lucene.NET/_build/results?buildId=2257&view=results
   https://dev.azure.com/lucene-net-temp2/Lucene.NET/_build/results?buildId=1014&view=results
   https://dev.azure.com/lucene-net-temp3/Lucene.NET/_build/results?buildId=850&view=results
   https://dev.azure.com/lucene-net-temp4/Main/_build/results?buildId=273&view=results
   
   Note that I left the commit with the `[Repeat(25)]` attribute on this branch. It should be reverted before merging to master to restore the original behavior.
   
   --------
   
   I also improved the `TestIndexWriterOnJRECrash` test by using a socket instead of a temp file to transfer the  process id to kill from the forked test to back to the original process. The original Java test crashed the JRE from the inside using a low level function, but .NET doesn't allow killing the current process from the inside so we needed 2-way communication between the 2 test processes. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1602506978


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   Pushed up a change that uses UninterruptableMonitor around a new lock object instead of using ConcurrentHashSet, to avoid the race conditions. ConcurrentHashSet becomes redundant here in that case. Since the hashset field is protected, I added some documentation to the fields explaining that any derived types that need to access them should lock on syncLock before accessing `m_staleFiles`.
   
   Also, what is our stance on field naming? Should we change `m_staleFiles` back to match Lucene's `staleFiles`? It doesn't seem to be consistent throughout the app. I personally would prefer to match Lucene, but let me know your thoughts. Renaming a protected field is a breaking change, so we should do that now if we're going to do it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2113312471

   Very nice! Do you think that maybe the thread-unsafety of FSDirectory (see my comment above about the race conditions) could be causing that? And maybe if .NET 5+ performs better than .NET FX, that is exacerbating it? I'll look at improving the thread safety here next and see if that can have better results. Because one very real possibility in this code (and AFAICT Java Lucene 4.8 too) under heavy load is that an fsync/FlushFileBuffers gets missed because of thread races.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2113877070

   > Very nice! Do you think that maybe the thread-unsafety of FSDirectory (see my comment above about the race conditions) could be causing that? And maybe if .NET 5+ performs better than .NET FX, that is exacerbating it? I'll look at improving the thread safety here next and see if that can have better results. Because one very real possibility in this code (and AFAICT Java Lucene 4.8 too) under heavy load is that an fsync/FlushFileBuffers gets missed because of thread races.
   
   Now that you mention it, it is possible that is the cause, (or a contributor) since we could be Fsyncing a partially written buffer to disk. While building a production quality implementation may take some time, if we locked all access to write, flush, and sync  on all files, it would make testing the scenario pretty simple. I will set that up and run the check.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "turowicz (via GitHub)" <gi...@apache.org>.
turowicz commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2247101054

   Is this moving forward?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on PR #938:
URL: https://github.com/apache/lucenenet/pull/938#issuecomment-2111472040

   I am going to do a check with the TestIndexWriterOnJRECrash test to try to confirm this is a fix for it. There is an issue with the test itself (something about the system property with the temp file name not being passed through) that I am working on a fix for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597522965


##########
src/Lucene.Net.Tests/Support/TestConcurrentHashSet.cs:
##########
@@ -0,0 +1,51 @@
+using Lucene.Net.Attributes;
+using Lucene.Net.Support;
+using NUnit.Framework;
+using System.Linq;
+using System.Threading.Tasks;
+
+namespace Lucene.Net
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    public class TestConcurrentHashSet

Review Comment:
   Unfortunately, the author of this collection didn't provide any tests. https://github.com/i3arnon/ConcurrentHashSet. But since it is based on `ConcurrentDictionary`, maybe we could also base the tests on the `ConcurrentDictionary` tests...?
   
   As you have noticed, it is also not fully implemented, otherwise it could be moved to J2N.Colllections.Concurrent with the rest of the collections.
   
   These issues may be relevant:
   
   - https://github.com/dotnet/runtime/issues/16443
   - https://github.com/dotnet/runtime/issues/39919
   
   I disagree with them about the benefits of this, though. `ConcurrentDictionary` doens't implment `ISet<T>` and therefore cannot be plugged into an API that accepts the interface. It also makes set operations such as `IntersectWith` hard to deal with.



##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -516,8 +519,53 @@ public static void ReThrowUnchecked(Exception th)
             }
         }
 
-        // LUCENENET specific: Fsync is pointless in .NET, since we are
-        // calling FileStream.Flush(true) before the stream is disposed
-        // which means we never need it at the point in Java where it is called.
+        public static void Fsync(FileSystemInfo fileToSync, bool isDir)

Review Comment:
   With `FileSystemInfo`, there is no reason for the boolean parameter. We can check whether a `DirectoryInfo` or `FileInfo` object is passed by using `fileToSync is DirectoryInfo` or `fileToSync is FileInfo`.
   
   That said, it would make more sense to pass a string and boolean here as they did in Java because we can remove the allocations of creating these objects in .NET. All of the .NET APIs accept a string instead of a `FileSystemInfo`. See #832.



##########
src/Lucene.Net/Support/IO/WindowsFsyncSupport.cs:
##########
@@ -0,0 +1,119 @@
+using System;
+using System.IO;
+using System.Runtime.InteropServices;
+
+namespace Lucene.Net.Support.IO
+{
+    /*
+     * Licensed to the Apache Software Foundation (ASF) under one or more
+     * contributor license agreements.  See the NOTICE file distributed with
+     * this work for additional information regarding copyright ownership.
+     * The ASF licenses this file to You under the Apache License, Version 2.0
+     * (the "License"); you may not use this file except in compliance with
+     * the License.  You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    public static class WindowsFsyncSupport

Review Comment:
   I noticed that other projects follow a consistent naming convention ending in a `.<platform>.cs` suffix. I have seen some projects even make a distinction between `Windows` and `Win32`. I believe this would go under `Win32`, though.
   
   I think we should consider how adding more native calls would work and follow Microsoft's lead on this. Maybe we should put all native code in a `Support/Native` folder and use nested classes like `Interop.Win32.FSync`. Thoughts?



##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   This is not a threadsafe implementation - this operation should be atomic. Please see the implementation of `UnionWith` to lock the set for updating. Do note, however, that implementation is untested like the rest of the collection (other than not known to fail in integration tests).



##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -516,8 +519,53 @@ public static void ReThrowUnchecked(Exception th)
             }
         }
 
-        // LUCENENET specific: Fsync is pointless in .NET, since we are
-        // calling FileStream.Flush(true) before the stream is disposed
-        // which means we never need it at the point in Java where it is called.
+        public static void Fsync(FileSystemInfo fileToSync, bool isDir)
+        {
+            // LUCENENET NOTE: there is a bug in 4.8 where it tries to fsync a directory on Windows,
+            // which is not supported in OpenJDK. This change adopts the latest Lucene code as of 9.10
+            // and only fsyncs directories on Linux and macOS.
+
+            // If the file is a directory we have to open read-only, for regular files we must open r/w for
+            // the fsync to have an effect.
+            // See http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/
+            if (isDir && Constants.WINDOWS)
+            {
+                // opening a directory on Windows fails, directories can not be fsynced there
+                if (System.IO.Directory.Exists(fileToSync.FullName) == false)
+                {
+                    // yet do not suppress trying to fsync directories that do not exist
+                    throw new DirectoryNotFoundException($"Directory does not exist: {fileToSync}");
+                }
+                return;
+            }
+
+            try
+            {
+                // LUCENENET specific: was: file.force(true);
+                // We must call fsync on the parent directory, requiring some custom P/Invoking
+                if (Constants.WINDOWS)
+                {
+                    WindowsFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+                else
+                {
+                    PosixFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+            }
+            catch (Exception e) when (e.IsIOException())
+            {
+                if (isDir)
+                {
+                    Debug.Assert((Constants.LINUX || Constants.MAC_OS_X) == false,

Review Comment:
   Please use our `Debugging.Assert` method here. For performance reasons, it must go inside of a `if (Debugging.AssertsEnabled)` block to be as cheap as possible in production while still retaining the ability to assert during testing.
   
   See https://github.com/apache/lucenenet/issues/346#issuecomment-696780561.
   
   Please note that we don't want this call to do any string formatting or concatenation inline, as that directly impacts the runtime performance of the code. So, in this case, we would want to pass the `e` as a parameter to `Debugging.Assert` and use a format string to insert it. The Assert method doesn't do any string manipulation unless the assert fails. And although it affects readability, we should eliminate the string concatenation here.



##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -516,8 +519,53 @@ public static void ReThrowUnchecked(Exception th)
             }
         }
 
-        // LUCENENET specific: Fsync is pointless in .NET, since we are
-        // calling FileStream.Flush(true) before the stream is disposed
-        // which means we never need it at the point in Java where it is called.
+        public static void Fsync(FileSystemInfo fileToSync, bool isDir)
+        {
+            // LUCENENET NOTE: there is a bug in 4.8 where it tries to fsync a directory on Windows,
+            // which is not supported in OpenJDK. This change adopts the latest Lucene code as of 9.10
+            // and only fsyncs directories on Linux and macOS.
+
+            // If the file is a directory we have to open read-only, for regular files we must open r/w for
+            // the fsync to have an effect.
+            // See http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/
+            if (isDir && Constants.WINDOWS)
+            {
+                // opening a directory on Windows fails, directories can not be fsynced there
+                if (System.IO.Directory.Exists(fileToSync.FullName) == false)
+                {
+                    // yet do not suppress trying to fsync directories that do not exist
+                    throw new DirectoryNotFoundException($"Directory does not exist: {fileToSync}");
+                }
+                return;
+            }
+
+            try
+            {
+                // LUCENENET specific: was: file.force(true);
+                // We must call fsync on the parent directory, requiring some custom P/Invoking
+                if (Constants.WINDOWS)
+                {
+                    WindowsFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+                else
+                {
+                    PosixFsyncSupport.Fsync(fileToSync.FullName, isDir);
+                }
+            }
+            catch (Exception e) when (e.IsIOException())
+            {
+                if (isDir)
+                {
+                    Debug.Assert((Constants.LINUX || Constants.MAC_OS_X) == false,

Review Comment:
   Please use our `Debugging.Assert` method here. For performance reasons, it must go inside of a `if (Debugging.AssertsEnabled)` block to be as cheap as possible in production.
   
   See https://github.com/apache/lucenenet/issues/346#issuecomment-696780561.



##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))
+            {
+                Clear();
+                return;
+            }
+
+            foreach (var item in other)
+            {
+                TryRemove(item);

Review Comment:
   Please see the optimizations that Microsoft added at: https://github.com/dotnet/runtime/blob/7ce5d0fdf728bb2ae213541b17c2e489d2b06d09/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/HashSet.cs#L501. We can probably improve this implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597690668


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   Technically this is thread-safe, but you're right that it's not atomic. This method only calls either `Clear` or `TryRemove`, both of which are thread-safe. Since it is non-atomic, it would allow concurrent adds/etc in the middle of the iteration, which as an aside theoretically could be preferred in some use cases (which we don't have, which is why this is an aside) where reducing duration or scope of locking of add operations is preferred to atomic bulk removal, such as in a caching case where the cost of a cache miss is lower than the cost of atomic locking. But that's neither here nor there as there's a bigger concurrency concern here...
   
   Even if we make this method atomic, it won't solve the problem of ensuring consistency of the stale files hashset in `FSDirectory`. Example:
   
   1. Initial state of stale files is "a", "b", and "c"
   2. Thread 1 is doing some operation with the index output on "b"
   3. Thread 2 enters Sync and copies the stale files into `toSync`
   4. Thread 2 fsyncs "a" and "b", is about to fsync "c"
   5. Thread 1 completes its write to "b" and `OnIndexOutputClosed` is called, which doesn't add "b" into stale files because it already exists
   6. Thread 2 fsyncs "c" and calls `ExceptWith` (atomic or not), removing a-c
   7. "b" is not fsynced for the second write even if Thread 1 calls Sync again because it's now been removed from the set
   
   I think if there was a "clear-and-return-all-removed" method that would whack this mole, but then another pops up with the `DeleteFile` method, which could result in this problem:
   
   1. Initial state of stale files is "a", "b", and "c"
   2. Thread 1 enters Sync and copies (even with simultaneous clearing) all entries into `toSync`
   3. Thread 2 enters DeleteFile and deletes the file "b" on disk
   4. Thread 1 fsyncs and throws an exception because "b" no longer exists
   5. Thread 2 removes "b" from stale files
   
   I don't think there's any amount of concurrent-izing this hashset that avoids these potential race conditions. The only proper solution would be to synchronize around the entire hashset, at which point we might as well just use the BCL non-concurrent HashSet instead. Let me know what you think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1597692468


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))
+            {
+                Clear();
+                return;
+            }
+
+            foreach (var item in other)
+            {
+                TryRemove(item);

Review Comment:
   Also this raises that we should probably add an optimization to `FSDirectory.Sync` that doesn't try to do its logic if the stale files (or `toSync` copy) hashset is empty... then we wouldn't even get to the `ExceptWith` optimization in this case anyways.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Restore fsync behavior in FSDirectory via P/Invoke, #933 [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #938:
URL: https://github.com/apache/lucenenet/pull/938#discussion_r1602508557


##########
src/Lucene.Net/Support/ConcurrentHashSet.cs:
##########
@@ -753,7 +753,16 @@ private void CopyToItems(T[] array, int index)
 
         public void ExceptWith(IEnumerable<T> other)
         {
-            throw new NotImplementedException();
+            if (ReferenceEquals(this, other))

Review Comment:
   Well, nevermind on the names bit, I encountered the tests that ensure we name them correctly. Fixing now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org