You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "NehanPathan (via GitHub)" <gi...@apache.org> on 2025/09/10 10:57:12 UTC

[PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

NehanPathan opened a new pull request, #1182:
URL: https://github.com/apache/lucenenet/pull/1182

   @paulirwin @NightOwl888 
   ---
   **Fixes:**
   Fixes #1181
   
   **Summary of the changes:**
   Implemented `IAsyncReplicator` and async methods in `ReplicationClient` for non-blocking replication operations.
   
   ---
   
   ## Description
   
   This PR introduces an async Task-based API for the replication client, allowing operations such as checking for updates, obtaining files, and releasing sessions to be executed asynchronously.
   
   Key changes:
   
   * Added `IAsyncReplicator` interface.
   * Updated `HttpReplicator` to implement async versions of the operations (`CheckForUpdateAsync`, `ObtainFileAsync`, `ReleaseAsync`, `PublishAsync`).
   * Added async helper methods in `HttpReplicator` to wrap HTTP requests using `HttpClient`.
   * `ReplicationClient` now exposes async methods that call into `IAsyncReplicator`.
   
   This avoids synchronous HTTP calls that could deadlock or cause performance issues, while keeping `IReplicationHandler` synchronous, as Lucene.NET APIs currently do not have async equivalents.
   
   **Additional context:**
   This implementation has been tested and works successfully when using `UpdateNowAsync` in the GSoC project by referencing the Lucene.NET repository in the GSoC extensions project.
   
   ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2339984262


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   Thanks for clarifying! You’re right — marking sync methods [Obsolete] was premature. Sync APIs are still important for backward compatibility and debugging against Java. Since ReplicationClient can already be extended and mocked via inheritance, we don’t need separate interfaces for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "Shazwazza (via GitHub)" <gi...@apache.org>.
Shazwazza commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2760132164


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   Actually - instead of doing that (or maybe that is done as a private ctor or similar). We can have a static factory method to return either IReplicator or IAsyncReplicator:
   
   * ReplicationClient.CreateAsyncClient(...)
   * ReplicationClient.CreateClient(...)
   
   That way, you can separate concerns and return the correct interface. Then IAsyncReplicator probably should not inherit from IReplicator and you can keep them separated, but the implementation can remain within the same class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338352767


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -217,6 +240,31 @@ private string QueryString(string request, params string[] parameters)
                 .Join("&", parameters.Select(WebUtility.UrlEncode).InPairs((key, val) => string.Format("{0}={1}", key, val))));
         }
 
+        // Add this property so subclasses can access the HttpClient instance
+        protected HttpClient Client => httpc;
+
+        // BuildUrl helpers (mirror the QueryString overloads)
+        protected virtual string BuildUrl(string action, string[] parameters)

Review Comment:
   @NightOwl888
   
   Sorry — initially while implementing async, I had put the ExecuteGetAsync methods in the subclass and used BuildUrl from the base class. Later I realized the sync ExecuteGet methods were already in HttpClientBase, so I shifted the async ones there as well for consistency. But I forgot to remove the extra BuildUrl wrappers (and the Client property), which are no longer needed since everything goes through QueryString inside HttpClientBase.
   
   I’ll clean those up now.
   
   thanks for that....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-3372299586

   I performed a force push on the feature/async-replicator-client branch to remove the merge commit from apache:master.
   
   Reason: The merge introduced changes from master that we do not want in this feature branch, and the branch already contains all the intended commits.
   
   This ensures the branch history is clean and only includes relevant feature work.
   
   if master branch changes are important to merge then pls let me know


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338631014


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -261,6 +279,101 @@ private void DoUpdate()
             }
         }
 
+        /// <summary>
+        /// Performs the async update logic, mirrors DoUpdate but uses IAsyncReplicator.
+        /// </summary>
+        private async Task DoUpdateAsync(CancellationToken cancellationToken)
+        {
+            if (asyncReplicator is null)
+                throw new InvalidOperationException("AsyncReplicator not initialized.");
+
+            SessionToken? session = null;

Review Comment:
   @NehanPathan 
   
   `#nullable enable` can be turned off again using `#nullable restore`. While it is preferable to solve all of the nullable issues rather than ignore them, this at least lets us put them off until they can be considered separately from ongoing work. I would highly recommend that the new code we are adding is fenced between `#nullable enable` and `#nullable restore` blocks, which will save us the work of having to do it all later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-4285381918

   @paulirwin 
   Thanks for taking this forward....
   
   I’ll start through the Copilot comments (especially around async flow, disposal, and streaming), and will align with your direction on the constructor approach.
   
   Let me know if there are specific areas you’d prefer I focus on.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r3067433969


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   @Shazwazza Picking this back up again... I ended up going with your suggestion of having IAsyncReplicator inherit from IReplicator, and do a type check. I really like this approach. It also lets us easily wrap the sync methods in Task-based code if you pass in a non-IAsyncReplicator instance, while still letting those methods operate without throwing (they just wouldn't actually be asynchronous). Let me know your thoughts on my latest commit: https://github.com/apache/lucenenet/pull/1182/commits/a6972913fe87325a4fd9486936f3852ccde0f441



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-3372403420

   I’m seeing the check-editorconfig CI fail due to a “final newline expected” error. Locally, git diff --check doesn’t show any issues, so I suspect it might be related to line endings (CRLF vs LF) or some subtle trailing whitespace.
   
   Could you please guide me on the safest way to fix this, such as how to identify files with extra trailing whitespace or missing newlines, so the CI passes without affecting other files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338440760


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -261,6 +279,101 @@ private void DoUpdate()
             }
         }
 
+        /// <summary>
+        /// Performs the async update logic, mirrors DoUpdate but uses IAsyncReplicator.
+        /// </summary>
+        private async Task DoUpdateAsync(CancellationToken cancellationToken)
+        {
+            if (asyncReplicator is null)
+                throw new InvalidOperationException("AsyncReplicator not initialized.");
+
+            SessionToken? session = null;

Review Comment:
   @paulirwin @NightOwl888 
   can you pls suggest better way to adjust whole file after adding #nullable enable
   beacause once i add that and try to resolve fields
   it actually 2x or 3x my warnings after every build ie from 32 -> 84 -> 153 warnings



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-5029088513

   @paulirwin 
   I reviewed the latest changes and they look good to me. 
   
   The updated architecture matches the direction we discussed earlier, with IAsyncReplicator remaining in the core library while the async replication client is moved out to the Extensions project. The cleanup around removing the async ReplicationClient implementation looks complete, and I didn't notice any dangling references or inconsistencies after the refactoring.
   Thanks....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r3135259165


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   @Shazwazza I did, but perhaps I am/was confused. It seems from the name that those would not return `IReplicator`/`IAsyncReplicator` as you originally mentioned, but maybe you meant new `IReplicationClient`/`IAsyncReplicationClient` interfaces? A method named `ReplicationClient.CreateClient` seems like it should not return an `IReplicator` (which, naming clarity or the lack thereof aside, is not the same thing as a replication client) but instead something like `IReplicationClient` (which does not yet exist).
   
   If we make the ReplicationClient constructor private so that it is only called by the factory methods, that would be a breaking change, but I think we are probably okay with that.
   
   I would be fine with having methods like you mention, if they return new `IReplicationClient` and `IAsyncReplicationClient` interfaces, and take `IReplicator` / `IAsyncReplicator` parameters respectively, and make the constructor(s) of `ReplicationClient` private. Then ReplicationClient would essentially be an opaque implementation of both interfaces, or could be a base class to share core functionality with two different internal implementations, and all use of replication clients would go through the interfaces.
   
   Let me know if that is what you intended, or if you meant something else. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338352767


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -217,6 +240,31 @@ private string QueryString(string request, params string[] parameters)
                 .Join("&", parameters.Select(WebUtility.UrlEncode).InPairs((key, val) => string.Format("{0}={1}", key, val))));
         }
 
+        // Add this property so subclasses can access the HttpClient instance
+        protected HttpClient Client => httpc;
+
+        // BuildUrl helpers (mirror the QueryString overloads)
+        protected virtual string BuildUrl(string action, string[] parameters)

Review Comment:
   @NightOwl888
   
   Sorry — initially while implementing async, I had put the ExecuteGetAsync methods in the subclass ie HttpReplicator and used BuildUrl from the base class ie HttpClientBase. Later I realized the sync ExecuteGet methods were already in HttpClientBase, so I shifted the async ones there as well for consistency. But I forgot to remove the extra BuildUrl wrappers (and the Client property), which are no longer needed since everything goes through QueryString inside HttpClientBase.
   
   I’ll clean those up now.
   
   thanks for that....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-3564943008

   I have rebased this PR on latest master. Had to do #1221 to fix the build error due to an editorconfig-checker update. I'll look at resolving the remaining comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-4283850041

   @NehanPathan I'll review Copilot's feedback here (as well as run it through a few other models) and will update the PR with any needed changes. We'll proceed with my latest commit for the constructor approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin merged PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2336629792


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +200,29 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[] parameters, CancellationToken cancellationToken)
+        {
+            var url = BuildUrl(action, parameters);

Review Comment:
   The other methods call EnsureOpen first. Please do so here.



##########
src/Lucene.Net.Replicator/IAsyncReplicator.cs:
##########
@@ -0,0 +1,35 @@
+using System.IO;
+using System.Threading;
+using System.Threading.Tasks;
+
+namespace Lucene.Net.Replicator
+{
+    /// <summary>
+    /// Async version of <see cref="IReplicator"/> for non-blocking replication operations.
+    /// </summary>
+    public interface IAsyncReplicator
+    {
+        /// <summary>
+        /// Check whether the given version is up-to-date and returns a
+        /// <see cref="SessionToken"/> which can be used for fetching the revision files.
+        /// </summary>
+        /// <param name="currentVersion">Current version of the index.</param>
+        /// <param name="cancellationToken">Cancellation token.</param>
+        Task<SessionToken?> CheckForUpdateAsync(string currentVersion, CancellationToken cancellationToken = default);

Review Comment:
   Add `#nullable enable` to this file to resolve these warnings.



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -261,6 +279,101 @@ private void DoUpdate()
             }
         }
 
+        /// <summary>
+        /// Performs the async update logic, mirrors DoUpdate but uses IAsyncReplicator.
+        /// </summary>
+        private async Task DoUpdateAsync(CancellationToken cancellationToken)
+        {
+            if (asyncReplicator is null)
+                throw new InvalidOperationException("AsyncReplicator not initialized.");
+
+            SessionToken? session = null;

Review Comment:
   Please add `#nullable enable` to this file to resolve these warnings, and fix any nullability issues that might arise as a result.



##########
src/Lucene.Net.Replicator/Http/HttpReplicator.cs:
##########
@@ -106,5 +109,95 @@ public virtual void Release(string sessionId)
             // do not remove this call: as it is still validating for us!
             DoAction<object>(response, () => null);
         }
+
+        #region Async methods (IAsyncReplicator)
+
+        /// <summary>
+        /// Checks for updates at the remote host asynchronously.
+        /// </summary>
+        /// <param name="currentVersion">The current index version.</param>
+        /// <param name="cancellationToken">Cancellation token.</param>
+        /// <returns>
+        /// A <see cref="SessionToken"/> if updates are available; otherwise, <c>null</c>.
+        /// </returns>
+        public async Task<SessionToken?> CheckForUpdateAsync(string currentVersion, CancellationToken cancellationToken = default)

Review Comment:
   Please add `#nullable enable` to the top of this file to resolve these warnings, and then if that exposes other nullability issues in the file, please fix them.



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   So, this is a brittle design. It means that you have to know which constructor was used in order to know which methods to call and not get an exception at runtime. I think we should break this type apart.
   
   Let me know your thoughts on this:
   1. Refactor out a common ReplicationClientBase abstract class with any common methods/fields
   2. Make this type inherit from the base class. Leave `// LUCENENET: ` comments where methods were refactored into the base class, to aid future porting efforts. Move the async methods into item 3 below.
   3. Introduce a new AsyncReplicationClient that inherits from ReplicationClientBase, and only has the async methods. This way you can know that it will have an async replicator field.
   
   Thoughts?



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -468,6 +695,25 @@ public virtual void UpdateNow()
             }
         }
 
+        /// <summary>
+        /// Executes the update operation asynchronously immediately, regardless if an update thread is running or not.
+        /// </summary>
+        public virtual async Task UpdateNowAsync(CancellationToken cancellationToken = default)
+        {
+            EnsureOpen();
+
+            // Acquire the same update lock to prevent concurrent updates
+            updateLock.Lock();

Review Comment:
   Question for @NightOwl888: is this async-safe?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338433727


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +200,29 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[] parameters, CancellationToken cancellationToken)

Review Comment:
   okay i'll try to update that
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2721393548


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   @Shazwazza Do you have thoughts on the options above, since you've worked with Replicator? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-4482090372

   I picked this back up again, trying to finish it up and ensure all the Copilot feedback is addressed properly, and discovered several likely bugs, particularly around synchronization (reentrant vs non-reentrant primitives, async deadlocks, disposal, task completion exceptions, etc). Additionally, "doing this right" while trying to adhere to some of the design principles discussed here (i.e. having sync and async versions in the same class) is going to result in a maintenance burden, I'm afraid, because it can't simply be done the same way "but just made async."
   
   For example, the `public synchronized void close()` in Java calls `public synchronized void stopUpdateThread()`, which is allowed because Java's synchronized keyword is reentrant. Our UninterruptableMonitor is reentrant too, but it can't be used around an `await` since it might resume on a different thread. So now our Dispose/StopAsyncUpdateLoop methods have to dance around this and it seems from even the most generous of readings that this code is bug-prone now. Not to mention having to consider thread interrupt stuff.
   
   My current thinking has gone back to thinking that we would actually be better served by splitting out a separate AsyncReplicationClient class, or even moving that support to the extensions project as a Microsoft.Extensions.Hosting BackgroundService. That way we can keep ReplicationClient very close to upstream (with no breaking changes for users), still support IAsyncReplicator in our Replicator library and its implementation in HttpReplicator as primitives for others to use if needed, but allow for our async-based ReplicationClient implementation to follow a different path.
   
   I could go either way on whether to make it another class in our Replicator library, or to move to Extensions. Pros for each:
   
   - **Option A: Separate Class in Lucene.Net.Replicator**:
     - Can see implementations side by side, and possibly even refactor out a common base class for some things
     - We can note in comments that future porting efforts need to update both classes
     - Will be easier to port upstream changes to each if they both look similar to upstream's ReplicationClient (just one with async-colored glasses)
   - **Option B: Move to Extensions**:
     - Can use Microsoft.Extensions.Hosting's BackgroundService for a more modern-.NET idiomatic approach to running code in a background loop, which would likely result in significantly less code (no need to synchronize start/stop state, etc)
     - Can diverge from upstream and follow its own path
     - Supports Microsoft.Extensions.DependencyInjection
     - Would pair well with @NehanPathan's other extensions work (I still need to finish merging that btw)
   
   And to clarify and emphasize what I said above: IAsyncReplicator and HttpReplicator's implementation of it would remain in this PR if we choose option B. I'm currently leaning towards that, but open to option A too if there's a strong consensus for that. 
   
   Also, it should be noted, async support for a replication client was primarily attempted out of consistency with the rest of our async support for replication. It's typically not strictly necessary, since you don't need to worry about hogging a thread for a background job which likely just has one thread anyways. Since this is proving to be more complex, difficult, and disruptive to future porting efforts than anticipated, I think we can pivot to Option B which would allow for a more modern IHostedService approach, still support the asynchronous code, and keep it out of the way for future porting efforts and bug fixes.
   
   @NightOwl888 @NehanPathan @Shazwazza Please let me know your thoughts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2337789009


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -217,6 +240,31 @@ private string QueryString(string request, params string[] parameters)
                 .Join("&", parameters.Select(WebUtility.UrlEncode).InPairs((key, val) => string.Format("{0}={1}", key, val))));
         }
 
+        // Add this property so subclasses can access the HttpClient instance
+        protected HttpClient Client => httpc;
+
+        // BuildUrl helpers (mirror the QueryString overloads)
+        protected virtual string BuildUrl(string action, string[] parameters)

Review Comment:
   I am curious why are we wrapping the private `QueryString()` method and making it protected, here?



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -217,6 +240,31 @@ private string QueryString(string request, params string[] parameters)
                 .Join("&", parameters.Select(WebUtility.UrlEncode).InPairs((key, val) => string.Format("{0}={1}", key, val))));
         }
 
+        // Add this property so subclasses can access the HttpClient instance
+        protected HttpClient Client => httpc;
+
+        // BuildUrl helpers (mirror the QueryString overloads)
+        protected virtual string BuildUrl(string action, string[] parameters)
+        {
+            // QueryString has signature: QueryString(string request, params string[] parameters)
+            return QueryString(action, parameters);
+        }
+
+        protected virtual string BuildUrl(

Review Comment:
   I am curious why are we wrapping the private `QueryString()` method and making it protected, here?



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -262,6 +310,37 @@ public virtual Stream GetResponseStream(HttpResponseMessage response, bool consu
             return result;
         }
 
+        /// <summary>

Review Comment:
   Note that above in `GetResponseStream()`, in `net8.0` there is now a synchronous overload of `ReadAsStream()` that we should be calling here, instead.
   
   Please add a new feature `FEATURE_HTTPCONTENT_READASSTREAM` to the root `Directory.Build.targets`.



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   I suppose it is unlikely a user will require both async and synchronous methods at the same time. And it would clarify the `StartAsyncUpdateLoop` vs `StartUpdateThread()` logic. Although, I think both of those could be potentially combined.
   
   I am hesitant to get on board with having completely separate implementations, though. That is not typically how concrete components in the BCL evolve. Usually, the synchronous and asynchronous methods exist side by side.



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -262,6 +310,37 @@ public virtual Stream GetResponseStream(HttpResponseMessage response, bool consu
             return result;
         }
 
+        /// <summary>
+        /// Internal utility: input stream of the provided response asynchronously.
+        /// </summary>
+        /// <exception cref="IOException"></exception>
+        public virtual async Task<Stream> GetResponseStreamAsync(HttpResponseMessage response, CancellationToken cancellationToken = default)
+        {
+        #if NET8_0_OR_GREATER

Review Comment:
   Please add a new feature `FEATURE_HTTPCONTENT_READASSTREAM_CANCELLATIONTOKEN` to the root `Directory.Build.targets` instead of using `NET8_0_OR_GREATER`.



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -262,6 +310,37 @@ public virtual Stream GetResponseStream(HttpResponseMessage response, bool consu
             return result;
         }
 
+        /// <summary>
+        /// Internal utility: input stream of the provided response asynchronously.
+        /// </summary>
+        /// <exception cref="IOException"></exception>
+        public virtual async Task<Stream> GetResponseStreamAsync(HttpResponseMessage response, CancellationToken cancellationToken = default)
+        {
+        #if NET8_0_OR_GREATER
+            Stream result = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
+        #else
+            Stream result = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
+        #endif
+            return result;
+        }
+
+        /// <summary>
+        /// Internal utility: input stream of the provided response asynchronously, which optionally
+        /// consumes the response's resources when the input stream is exhausted.
+        /// </summary>
+        /// <exception cref="IOException"></exception>
+        public virtual async Task<Stream> GetResponseStreamAsync(HttpResponseMessage response, bool consume, CancellationToken cancellationToken = default)
+        {
+        #if NET8_0_OR_GREATER

Review Comment:
   Please add a new feature `FEATURE_HTTPCONTENT_READASSTREAM_CANCELLATIONTOKEN` to the root `Directory.Build.targets` instead of using `NET8_0_OR_GREATER`.



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +200,29 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[] parameters, CancellationToken cancellationToken)

Review Comment:
   Perhaps we should fix `ExecutePost()` in this PR and roll #1176 in with this? We will just have to add an `ExecutePostAsync()` method later, anyway.



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -468,6 +695,25 @@ public virtual void UpdateNow()
             }
         }
 
+        /// <summary>
+        /// Executes the update operation asynchronously immediately, regardless if an update thread is running or not.
+        /// </summary>
+        public virtual async Task UpdateNowAsync(CancellationToken cancellationToken = default)
+        {
+            EnsureOpen();
+
+            // Acquire the same update lock to prevent concurrent updates
+            updateLock.Lock();

Review Comment:
   According to ChatGPT, no this is not. Calling async code while using a synchronous lock can lead to deadlocks.
   
   Do note that in other places where the call is to `Task.Run()`, this is fine to do.
   
   ### Options
   
   1. Use `SemaphoreSlim` instead of `ReentrantLock`. Note that `SemaphoreSlim` is not re-entrant, though.
      - Safe across await.
      - Prevents reentrant calls, which may actually be a feature (forces clear lock discipline).
      - But if you really need recursion/reentrancy, this won’t do.
   2. Implement an async-compatible reentrant lock
      - This is tricky but possible.
      - You’d track:
         - Owning Thread or Task.Id
         - A recursion count
         - A `SemaphoreSlim` for waiting
      - On EnterAsync, if the current context already owns the lock, increment recursion count and continue immediately. Otherwise, wait.
      - On Exit, decrement recursion count and release only when it reaches 0.
   
   Using `SemaphoreSlim` is certainly easier, but I have no idea whether reentrancy is actually required by the design, or if that is just something that happened by accident because `ReentrantLock` was the primitive that was chosen for synchronization. It would be possible to refactor the design to provide protected methods that are not protected by `SemaphoreSlim` so subclasses can avoid reentrancy. It would be harder to implement subclasses, but possible. I am not sure whether that covers general callers, though.
   
   In the long run, implementing an `AsyncReentrantLock` could prove to be very useful for converting other parts of Lucene to be async, and would cause less pain for places where it is unknown whether reentrancy is a requirement. 
   
   Here is what such an implementation could look like. Note that I haven't tested this, it is straight out of ChatGPT. There may be a smarter way to do it than using a lock, but if we keep it, we would need to change `lock (this)` to `UninterruptableMonitor.Enter()` and `UninterruptableMonitor.Exit()` in a try/catch, to ensure it doesn't throw on thread interrupt.
   
   <details>
   
   <summary>AsyncReentrantLock - click to expand</summary>
   
   ```c#
   using System;
   using System.Collections.Concurrent;
   using System.Collections.Generic;
   using System.Threading;
   using System.Threading.Tasks;
   
   /// <summary>
   /// A lightweight async-compatible reentrant lock.
   /// </summary>
   public sealed class AsyncReentrantLock
   {
       private readonly SemaphoreSlim _semaphore = new(1, 1);
   
       // Owner + recursion count
       private int? _ownerThreadId;
       private int _recursionCount;
   
       // Track waiting requests
       private int _waitingCount;
   
       /// <summary> Number of threads/tasks currently waiting for the lock. </summary>
       public int WaitingCount => Volatile.Read(ref _waitingCount);
   
       /// <summary> True if the lock is currently held. </summary>
       public bool IsLocked => _ownerThreadId != null;
   
       /// <summary> Thread ID of the owner, or null if none. </summary>
       public int? OwnerThreadId => _ownerThreadId;
   
       /// <summary> Recursion depth of the current owner. </summary>
       public int RecursionCount => _recursionCount;
   
       // -----------------------
       // Sync API
       // -----------------------
       public void Lock(CancellationToken cancellationToken = default)
       {
           int currentId = Thread.CurrentThread.ManagedThreadId;
   
           lock (this)
           {
               if (_ownerThreadId == currentId)
               {
                   _recursionCount++;
                   return;
               }
           }
   
           Interlocked.Increment(ref _waitingCount);
           try
           {
               _semaphore.Wait(cancellationToken);
           }
           finally
           {
               Interlocked.Decrement(ref _waitingCount);
           }
   
           lock (this)
           {
               _ownerThreadId = currentId;
               _recursionCount = 1;
           }
       }
   
       // -----------------------
       // Async API
       // -----------------------
       public async Task LockAsync(CancellationToken cancellationToken = default)
       {
           int currentId = Thread.CurrentThread.ManagedThreadId;
   
           lock (this)
           {
               if (_ownerThreadId == currentId)
               {
                   _recursionCount++;
                   return;
               }
           }
   
           Interlocked.Increment(ref _waitingCount);
           try
           {
               await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
           }
           finally
           {
               Interlocked.Decrement(ref _waitingCount);
           }
   
           lock (this)
           {
               _ownerThreadId = currentId;
               _recursionCount = 1;
           }
       }
   
       // -----------------------
       // Unlock
       // -----------------------
       public void Unlock()
       {
           int currentId = Thread.CurrentThread.ManagedThreadId;
   
           lock (this)
           {
               if (_ownerThreadId != currentId)
                   throw new SynchronizationLockException("Current thread does not own the lock.");
   
               _recursionCount--;
   
               if (_recursionCount == 0)
               {
                   _ownerThreadId = null;
                   _semaphore.Release();
               }
           }
       }
   }
   ```
   
   </details>
   
   <details>
   
   <summary>AsyncReentrantLockTests - click to expand</summary>
   
   ```c#
   using NUnit.Framework;
   using System;
   using System.Threading;
   using System.Threading.Tasks;
   
   [TestFixture]
   public class AsyncReentrantLockTests
   {
       [Test]
       public void SyncLock_IsReentrant()
       {
           var l = new AsyncReentrantLock();
   
           l.Lock();
           Assert.That(l.IsLocked, Is.True);
           Assert.That(l.RecursionCount, Is.EqualTo(1));
   
           l.Lock();
           Assert.That(l.RecursionCount, Is.EqualTo(2));
   
           l.Unlock();
           Assert.That(l.RecursionCount, Is.EqualTo(1));
   
           l.Unlock();
           Assert.That(l.IsLocked, Is.False);
       }
   
       [Test]
       public async Task AsyncLock_IsReentrant()
       {
           var l = new AsyncReentrantLock();
   
           await l.LockAsync();
           Assert.That(l.IsLocked, Is.True);
   
           await l.LockAsync();
           Assert.That(l.RecursionCount, Is.EqualTo(2));
   
           l.Unlock();
           Assert.That(l.RecursionCount, Is.EqualTo(1));
   
           l.Unlock();
           Assert.That(l.IsLocked, Is.False);
       }
   
       [Test]
       public void ThrowsIfUnlockedByOtherThread()
       {
           var l = new AsyncReentrantLock();
           l.Lock();
   
           var ex = Assert.Throws<SynchronizationLockException>(() =>
           {
               var t = new Thread(() => l.Unlock());
               t.Start();
               t.Join();
           });
   
           Assert.That(ex.Message, Does.Contain("does not own"));
       }
   
       [Test]
       public async Task LockAsync_BlocksUntilReleased()
       {
           var l = new AsyncReentrantLock();
           l.Lock();
   
           var task = Task.Run(async () =>
           {
               await Task.Delay(100);
               l.Unlock();
           });
   
           await l.LockAsync(); // should block until Unlock is called
           Assert.That(l.IsLocked, Is.True);
   
           l.Unlock();
           await task;
       }
   
       [Test]
       public async Task WaitingCount_IsAccurate()
       {
           var l = new AsyncReentrantLock();
           l.Lock();
   
           var t1 = l.LockAsync();
           var t2 = l.LockAsync();
   
           await Task.Delay(50); // let them enqueue
   
           Assert.That(l.WaitingCount, Is.EqualTo(2));
   
           l.Unlock(); // release once
   
           await Task.WhenAll(t1, t2);
   
           l.Unlock(); // release final owner
       }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338267758


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   That said, I’m also flexible — if you both feel strongly about splitting into separate sync/async clients right now to minimize maintenance and testing overhead, I can adapt to that approach too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338305683


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -468,6 +695,25 @@ public virtual void UpdateNow()
             }
         }
 
+        /// <summary>
+        /// Executes the update operation asynchronously immediately, regardless if an update thread is running or not.
+        /// </summary>
+        public virtual async Task UpdateNowAsync(CancellationToken cancellationToken = default)
+        {
+            EnsureOpen();
+
+            // Acquire the same update lock to prevent concurrent updates
+            updateLock.Lock();

Review Comment:
   Yes, my bad — I was just trying to stay aligned with the sync version at that time.
   
   But after looking more carefully, I think we don’t actually need ReentrantLock for the sync version either. From what I see, inside doUpdate or doUpdateAsync there aren’t any reentrant scenarios, so it should be safe to use SemaphoreSlim in both cases.
   
   @NightOwl888  @paulirwin 
   Please correct me if I’m missing something or if there are still chances of reentrancy in the sync version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338622939


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   > Mark sync APIs with [Obsolete("Prefer async methods to avoid deadlocks.")] to gently guide users toward async.
   Long term: async becomes the default; sync kept only for backward compatibility.
   
   No, synchronous programming isn't going away any time soon. It is just in the space of HTTP servers, that it is no longer considered a good practice. Also, keep in mind that when debugging we will be comparing our implementation against the synchronous Java code. So, we should try to minimize changes to these methods.
   
   ---------------
   
   `ReplicationClient` is not sealed. It is designed to be inherited to provide additional functionality. So, in that regard, it is the abstraction and can provide the base implementation of both the synchronous and asynchronous sides. The question is since `ReplicationClient` can already be mocked by inheritance, do we need to add interfaces?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-4484020320

   @paulirwin 
   Thanks for digging deeper into this and explaining the concerns in more detail....
   
   After thinking through the synchronization and lifecycle side more carefully, the new direction makes more sense to me as well.
   
   Initially I was mostly looking at it from the perspective of “making async work” inside the existing `ReplicationClient`, and technically a lot of the locking issues can be worked around using approaches like `SemaphoreSlim`, avoiding `await` inside monitor locks, restructuring cleanup flow, etc. But the more I think about it, the more it feels like the complexity starts growing around the overall lifecycle management rather than any single async call itself.
   
   Especially with things like:
   
   * coordinating sync + async update loops
   * disposal and cancellation interactions
   * task completion/error handling
   * synchronization assumptions from the original Java design
   * and future maintenance/porting concerns
   
   Even if we solve those correctly now, it still adds quite a bit of long-term complexity to a class that was originally designed around a synchronous polling/thread model.
   
   Keeping `IAsyncReplicator` and the async support in `HttpReplicator` as reusable primitives while moving the hosted/background async behavior into Extensions using something like `BackgroundService` / `IHostedService` feels like a cleaner separation to me now as well. It also keeps the core replication library closer to upstream and probably makes future porting easier.
   
   I also think this could give us more flexibility if we eventually decide to evolve beyond polling-based replication loops toward more event-driven approaches without complicating the core replication client further.
   
   So overall I’m aligned with the new direction, and I’m happy to help with whichever approach we decide to move forward with.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338263549


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   agree with  @NightOwl888  suggestion
   I guess we should stick with one ReplicationClient that implements both IReplicator and IAsyncReplicator
   
   initially maintaining or testing it quite seems tough
   but i guess our future target is asyn only
   So may be 
   what we can do rightnow
   Mark sync APIs with [Obsolete("Prefer async methods to avoid deadlocks.")] to gently guide users toward async.
   Long term: async becomes the default; sync kept only for backward compatibility. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-3279751164

   @paulirwin @NightOwl888
   I’ve tried to cover most of the suggested changes, and the nullable warnings have also been resolved.
   Please let me know if I made any mistakes — I apologize in advance, as there were quite a few nullable warnings to address.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NightOwl888 (via GitHub)" <gi...@apache.org>.
NightOwl888 commented on PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#issuecomment-3372426885

   @NehanPathan 
   
   That error means that after all of the content in the file, there is no newline character. So, it just needs to be added.
   
   I know it is a bit confusing - remove all whitespace *except* at the end of the file, add a line break.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "Shazwazza (via GitHub)" <gi...@apache.org>.
Shazwazza commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2738028203


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   We can't do this: 
   
   > Remove synchronous replication support and lean in to async only. This would of course be a breaking change, but it would simplify the API.
   
   synchronous replication is still done and used by the IndexReplicationHandler and IndexAndTaxonomyReplicationHandler which just replicate between directories/indexes (locally and synchronously)
   
   I mean, i guess you can wrap the sync over async with Task.FromResult but this also adds some overhead - unless we use ValueTask which might be a thing to do anyways. This will be quite a big breaking change though.
   
   Why not do this:
   
   * Keep IAsyncReplicator (maybe use ValueTask if it fits the design)
   * Make IAsyncReplicator inherit from IReplicator
   * Keep only one ctor that accepts IReplicator - then type check to see if it implements IAsyncReplicator and store it in a nullable field
   * Add the async methods to the ReplicationClient as needed and within these methods, check if the IAsyncReplicator is null or not, if its not null, use it and return values. If it is null, get the result from the non-async IReplicator and return the value with Task.FromResult. Similarly, for the new async methods, perform this check but if the IAsyncReplicator field is null, either: throw a NotSupportedException, or return GetAwaiter().GetResult()
   
   This shouldn't introduce breaking changes and will support the async requirements since the user will now what their implementation is i.e. UpdateNow vs UpdateNowAsync



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r3468452874


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   Resolved via removing async support from ReplicationClient, per discussion below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "NehanPathan (via GitHub)" <gi...@apache.org>.
NehanPathan commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2338433727


##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +200,29 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[] parameters, CancellationToken cancellationToken)

Review Comment:
   okay i'll update that
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2551918542


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   There is another problem with the current approach of having two constructors: the constructors are ambiguous if you try to pass in a `HttpReplicator` without casting it to either interface first:
   
   <img width="1118" height="314" alt="screenshot-2025-11-21-at-20-19-15" src="https://github.com/user-attachments/assets/f154c390-fd9b-4b99-aa37-0040d0768780" />
   
   Our options include:
   1. Accept that you have to cast this before passing it in. I think this is a poor design choice for other reasons mentioned above, but also having to impose this on users is not ideal. Note that it's also technically a breaking change as-is, since we'd require people to cast HttpReplicator.
   2. Remove the IAsyncReplicator interface and move its async methods into IReplicator (so that it would have both async and sync methods), making any custom implementations either implement the async methods or throw NotImplementedException. This would be a breaking change because you'd make custom implementations add these async methods.
   3. A slight alternative to option 2 but with the same effect: make IAsyncReplicator inherit from IReplicator, and have ReplicationClient take only an IAsyncReplicator (which would now provide both methods). This is a breaking change since users of ReplicationClient with custom IReplicators would have to make their custom replicator implement IAsyncReplicator as well.
   4. Remove synchronous replication support and lean in to async only. This would of course be a breaking change, but it would simplify the API.
   5. Split out an AsyncReplicationClient as discussed above. AsyncReplicationClient would take an IAsyncReplicator; ReplicationClient would take a Replicator. This is the only non-breaking way to do this.
   
   @NightOwl888 thoughts? I could go for any of these except for option 1. I think if we're going to have any breaking changes we might as well do option 4 and just remove synchronous replication support to keep it simple, but I'm inclined to go for option 5 and just not have the breaking change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r2551886149


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -468,6 +695,25 @@ public virtual void UpdateNow()
             }
         }
 
+        /// <summary>
+        /// Executes the update operation asynchronously immediately, regardless if an update thread is running or not.
+        /// </summary>
+        public virtual async Task UpdateNowAsync(CancellationToken cancellationToken = default)
+        {
+            EnsureOpen();
+
+            // Acquire the same update lock to prevent concurrent updates
+            updateLock.Lock();

Review Comment:
   This seems fine as far as I can tell, but unit tests should expose any flaws. I'm working on tests now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r3113294048


##########
src/Lucene.Net.Replicator/Http/HttpReplicator.cs:
##########
@@ -109,5 +114,95 @@ public virtual void Release(string sessionId)
             // do not remove this call: as it is still validating for us!
             DoAction<object?>(response, () => null);
         }
+
+        #region Async methods (IAsyncReplicator)
+
+        /// <summary>
+        /// Checks for updates at the remote host asynchronously.
+        /// </summary>
+        /// <param name="currentVersion">The current index version.</param>
+        /// <param name="cancellationToken">Cancellation token.</param>
+        /// <returns>
+        /// A <see cref="SessionToken"/> if updates are available; otherwise, <c>null</c>.
+        /// </returns>
+        public async Task<SessionToken?> CheckForUpdateAsync(string? currentVersion, CancellationToken cancellationToken = default)
+        {
+            string[]? parameters = !string.IsNullOrEmpty(currentVersion)
+                ? new[] { ReplicationService.REPLICATE_VERSION_PARAM, currentVersion! } // [!]: verified above
+                : null;
+
+            using var response = await ExecuteGetAsync(
+                nameof(ReplicationService.ReplicationAction.UPDATE),
+                parameters,
+                cancellationToken: cancellationToken).ConfigureAwait(false);
+
+            return await DoActionAsync(response, async () =>
+            {
+                // ReSharper disable once AccessToDisposedClosure - DoActionAsync definitively returns after this lambda is invoked
+                using var inputStream = new DataInputStream(
+                    await GetResponseStreamAsync(response, cancellationToken).ConfigureAwait(false));
+
+                return inputStream.ReadByte() == 0 ? null : new SessionToken(inputStream);
+            }).ConfigureAwait(false);
+        }
+
+        /// <summary>
+        /// Obtains the given file from the remote host asynchronously.
+        /// </summary>
+        /// <param name="sessionId">The session ID.</param>
+        /// <param name="source">The source of the file.</param>
+        /// <param name="fileName">The file name.</param>
+        /// <param name="cancellationToken">Cancellation token.</param>
+        /// <returns>A <see cref="Stream"/> of the requested file.</returns>
+        public async Task<Stream> ObtainFileAsync(string sessionId, string source, string fileName, CancellationToken cancellationToken = default)
+        {
+            var response = await ExecuteGetAsync(
+                nameof(ReplicationService.ReplicationAction.OBTAIN),
+                ReplicationService.REPLICATE_SESSION_ID_PARAM, sessionId,
+                ReplicationService.REPLICATE_SOURCE_PARAM, source,
+                ReplicationService.REPLICATE_FILENAME_PARAM, fileName,
+                cancellationToken: cancellationToken).ConfigureAwait(false);
+
+            return await DoActionAsync(response, false,
+                    async () => await GetResponseStreamWithOwnershipAsync(response, cancellationToken).ConfigureAwait(false))
+                .ConfigureAwait(false);

Review Comment:
   In `ObtainFileAsync()`, the `HttpResponseMessage` is intentionally not disposed because the returned stream owns it, but if `DoActionAsync(...)` throws before returning the stream (e.g., due to an exception while creating the stream), `response` will leak. Wrap the `DoActionAsync` call in a try/catch (or use a `using` with a success flag) to ensure `response.Dispose()` happens on failure paths.
   ```suggestion
               try
               {
                   return await DoActionAsync(response, false,
                           async () => await GetResponseStreamWithOwnershipAsync(response, cancellationToken).ConfigureAwait(false))
                       .ConfigureAwait(false);
               }
               catch
               {
                   response.Dispose();
                   throw;
               }
   ```



##########
src/Lucene.Net/Util/IOUtils.cs:
##########
@@ -6,6 +6,7 @@
 using System;
 using System.Collections.Generic;
 using System.Diagnostics;
+using System.Diagnostics.CodeAnalysis;

Review Comment:
   `System.Diagnostics.CodeAnalysis` is imported but not used in this file, which can trigger an “unnecessary using” warning. Remove the unused using directive (or add the intended attribute usage) to keep builds clean.
   ```suggestion
   
   ```



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -430,7 +557,125 @@ public virtual bool IsUpdateThreadAlive
                 UninterruptableMonitor.Enter(syncLock);
                 try
                 {
-                    return updateThread != null && updateThread.IsAlive;
+                    return updateThread is not null && updateThread.IsAlive;
+                }
+                finally
+                {
+                    UninterruptableMonitor.Exit(syncLock);
+                }
+            }
+        }
+
+        /// <summary>
+        /// Start a periodic async update loop with the specified interval in milliseconds.
+        /// </summary>
+        /// <param name="intervalInMilliseconds">Interval between updates.</param>
+        public virtual void StartAsyncUpdateLoop(long intervalInMilliseconds)
+        {
+            UninterruptableMonitor.Enter(syncLock);
+            try
+            {
+                EnsureOpen();
+
+                if (asyncUpdateTask != null && !asyncUpdateTask.IsCompleted)
+                    throw IllegalStateException.Create("Async update loop is already running. Stop it first.");
+
+                asyncUpdateCts = new CancellationTokenSource();
+                CancellationToken ct = asyncUpdateCts.Token;
+
+                asyncUpdateTask = Task.Run(async () =>
+                {
+                    while (!ct.IsCancellationRequested)
+                    {
+                        try
+                        {
+                            await updateLock.WaitAsync(ct).ConfigureAwait(false);
+                            try
+                            {
+                                await DoUpdateAsync(ct).ConfigureAwait(false);
+                            }
+                            finally
+                            {
+                                updateLock.Release();
+                            }
+                        }
+                        catch (TaskCanceledException)
+                        {
+                            // Cancellation requested, exit loop gracefully
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            HandleUpdateException(ex);
+                        }
+
+                        try
+                        {
+                            await Task.Delay(TimeSpan.FromMilliseconds(intervalInMilliseconds), ct).ConfigureAwait(false);
+                        }
+                        catch (TaskCanceledException)
+                        {
+                            break; // exit loop if cancelled during delay

Review Comment:
   The new async update loop can be started via `StartAsyncUpdateLoop()`, but there’s no corresponding integration with `Dispose()`. If a caller forgets to call `StopAsyncUpdateLoop()`, the background task may continue running after the client is disposed. Consider updating the disposal/cleanup flow so disposing the client reliably cancels and awaits the async loop (and disposes the CTS) to prevent background work after disposal.
   ```suggestion
                   CancellationTokenSource cts = new CancellationTokenSource();
                   asyncUpdateCts = cts;
                   CancellationToken ct = cts.Token;
   
                   asyncUpdateTask = Task.Run(async () =>
                   {
                       try
                       {
                           while (!ct.IsCancellationRequested)
                           {
                               try
                               {
                                   EnsureOpen();
   
                                   await updateLock.WaitAsync(ct).ConfigureAwait(false);
                                   try
                                   {
                                       await DoUpdateAsync(ct).ConfigureAwait(false);
                                   }
                                   finally
                                   {
                                       updateLock.Release();
                                   }
                               }
                               catch (TaskCanceledException)
                               {
                                   // Cancellation requested, exit loop gracefully
                                   break;
                               }
                               catch (ObjectDisposedException)
                               {
                                   // Client is being disposed, stop the background loop.
                                   break;
                               }
                               catch (AlreadyClosedException)
                               {
                                   // Client is closed/disposed, stop the background loop.
                                   break;
                               }
                               catch (Exception ex)
                               {
                                   HandleUpdateException(ex);
                               }
   
                               try
                               {
                                   await Task.Delay(TimeSpan.FromMilliseconds(intervalInMilliseconds), ct).ConfigureAwait(false);
                               }
                               catch (TaskCanceledException)
                               {
                                   break; // exit loop if cancelled during delay
                               }
                           }
                       }
                       finally
                       {
                           UninterruptableMonitor.Enter(syncLock);
                           try
                           {
                               if (ReferenceEquals(asyncUpdateCts, cts))
                               {
                                   asyncUpdateCts = null;
                               }
   
                               if (ReferenceEquals(asyncUpdateTask, Task.CurrentId.HasValue ? asyncUpdateTask : asyncUpdateTask))
                               {
                                   asyncUpdateTask = null;
                               }
                           }
                           finally
                           {
                               UninterruptableMonitor.Exit(syncLock);
                               cts.Dispose();
   ```



##########
src/Lucene.Net.Tests.Replicator/Http/HttpReplicatorTest.cs:
##########
@@ -147,6 +148,25 @@ public void TestBasic()
             assertEquals(2, int.Parse(reader.IndexCommit.UserData["ID"], NumberStyles.HexNumber));
         }
 
+        [Test]
+        [LuceneNetSpecific]
+        public async Task TestBasicAsync()
+        {
+            IReplicator replicator = new HttpReplicator(host, port, ReplicationService.REPLICATION_CONTEXT + "/s1", server.CreateHandler());
+            ReplicationClient client = new ReplicationClient(replicator, new IndexReplicationHandler(handlerIndexDir, null),

Review Comment:
   `TestBasicAsync()` creates a `ReplicationClient` (and underlying `HttpReplicator`/`HttpClient`) without disposing it. Leaked clients can keep sockets/handlers alive and cause test flakiness. Use a `using`/`try-finally` to dispose the client (and/or the replicator) as is done in `TestServerErrors`.
   ```suggestion
               using ReplicationClient client = new ReplicationClient(replicator, new IndexReplicationHandler(handlerIndexDir, null),
   ```



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +222,36 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[]? parameters, CancellationToken cancellationToken)
+        {
+            EnsureOpen();
+            var url = QueryString(action, parameters);
+            return httpc.GetAsync(url, cancellationToken);
+        }
+
+        /// <summary>
+        /// Execute a GET request asynchronously with up to 3 name/value parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(
+            string action,
+            string param1, string value1,
+            string? param2 = null, string? value2 = null,
+            string? param3 = null, string? value3 = null,
+            CancellationToken cancellationToken = default)
+        {
+            EnsureOpen();
+            var url = (param2 == null && param3 == null)
+                ? QueryString(action, param1, value1)
+                : QueryString(action,
+                    param1, value1,
+                    param2 ?? string.Empty, value2 ?? string.Empty,
+                    param3 ?? string.Empty, value3 ?? string.Empty);
+            return httpc.GetAsync(url, cancellationToken);
+        }

Review Comment:
   Same issue in the 3-parameter overload of `ExecuteGetAsync(...)`: calling `httpc.GetAsync(...)` will buffer the entire response body by default. This overload should also use `SendAsync` with `HttpCompletionOption.ResponseHeadersRead` to preserve streaming behavior for large replication files.



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -430,7 +557,125 @@ public virtual bool IsUpdateThreadAlive
                 UninterruptableMonitor.Enter(syncLock);
                 try
                 {
-                    return updateThread != null && updateThread.IsAlive;
+                    return updateThread is not null && updateThread.IsAlive;
+                }
+                finally
+                {
+                    UninterruptableMonitor.Exit(syncLock);
+                }
+            }
+        }
+
+        /// <summary>
+        /// Start a periodic async update loop with the specified interval in milliseconds.
+        /// </summary>
+        /// <param name="intervalInMilliseconds">Interval between updates.</param>
+        public virtual void StartAsyncUpdateLoop(long intervalInMilliseconds)
+        {
+            UninterruptableMonitor.Enter(syncLock);
+            try
+            {
+                EnsureOpen();
+

Review Comment:
   `StartAsyncUpdateLoop()` doesn’t guard against the legacy `StartUpdateThread()` already running (and vice versa). As written, it’s possible to have both periodic mechanisms active, resulting in redundant back-to-back updates (serialized by `updateLock`) and unexpected behavior. Consider adding mutual-exclusion checks (e.g., throw if `IsUpdateThreadAlive` when starting async loop, and throw if `IsAsyncUpdateLoopAlive` when starting the thread).
   ```suggestion
   
                   if (updateThread is not null && updateThread.IsAlive)
                       throw IllegalStateException.Create("Update thread is already running. Stop it first.");
   ```



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -244,20 +254,137 @@ private void DoUpdate()
             try
             {
                 if (notify && !disposed)
-                { // no use to notify if we are closed already
+                {
+                    // no use to notify if we are closed already
                     // LUCENENET specific - pass the copiedFiles as read only
                     handler.RevisionReady(session.Version, session.SourceFiles, Collections.AsReadOnly(copiedFiles), sourceDirectory);
                 }
             }
             finally
             {
                 IOUtils.Dispose(sourceDirectory.Values);
-                //TODO: Resharper Message, Expression is always true -> Verify and if so then we can remove the null check.
+
+                // LUCENENET specific: removed redundant check. Code above either ensures it is not null (by returning early) or an exception is thrown.
+                // if (session != null)
+                // {
+                    factory.CleanupSession(session.Id);
+                // }
+            }
+        }
+
+        /// <summary>
+        /// Performs the async update logic. When the replicator implements
+        /// <see cref="IAsyncReplicator"/>, native async methods are used;
+        /// otherwise, the synchronous <see cref="IReplicator"/> methods are
+        /// called and their results wrapped.
+        /// </summary>
+        private async Task DoUpdateAsync(CancellationToken cancellationToken)
+        {
+            SessionToken? session = null;
+            var sourceDirectory = new Dictionary<string, Directory>();
+            var copiedFiles = new Dictionary<string, IList<string>>();
+            bool notify = false;
+
+            try
+            {
+                string? version = handler.CurrentVersion;
+
+                session = asyncReplicator is not null
+                    ? await asyncReplicator.CheckForUpdateAsync(version, cancellationToken).ConfigureAwait(false)
+                    : replicator.CheckForUpdate(version);
+
+                WriteToInfoStream($"DoUpdateAsync(): handlerVersion={version} session={session}");
+
+                if (session is null)
+                    return;
+
+                IDictionary<string, IList<RevisionFile>> requiredFiles = RequiredFiles(session.SourceFiles);
+
+                foreach (var pair in requiredFiles)
+                {
+                    string source = pair.Key;
+                    Directory directory = factory.GetDirectory(session.Id!, source);
+
+                    sourceDirectory.Add(source, directory);
+                    IList<string> cpFiles = new JCG.List<string>();
+                    copiedFiles.Add(source, cpFiles);
+
+                    foreach (var file in pair.Value)
+                    {
+                        cancellationToken.ThrowIfCancellationRequested();
+
+                        Stream? input = null;
+                        IndexOutput? output = null;
+                        try
+                        {
+                            input = asyncReplicator is not null
+                                ? await asyncReplicator.ObtainFileAsync(session.Id!, source, file.FileName, cancellationToken).ConfigureAwait(false)
+                                : replicator.ObtainFile(session.Id!, source, file.FileName);
+
+                            output = directory.CreateOutput(file.FileName, IOContext.DEFAULT);
+
+                            int numBytes;
+                            byte[] buffer = new byte[16384];
+                            while ((numBytes = await input.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) > 0)
+                            {
+                                output.WriteBytes(buffer, 0, numBytes);
+                            }

Review Comment:
   `DoUpdateAsync()` allocates a new 16KB buffer inside the inner file-copy loop (`byte[] buffer = new byte[16384];`), which adds avoidable allocations for each file. Since updates are serialized by `updateLock` and there’s already a `copyBuffer` field, consider reusing that buffer (or using `ArrayPool<byte>`) to reduce GC pressure during large replications.



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -173,20 +175,40 @@ protected virtual void ThrowKnownError(HttpResponseMessage response)
         }
 
         /// <summary>
-        /// <b>Internal:</b> Execute a request and return its result.
+        /// <b>Internal:</b> Execute a POST request with custom HttpContent and return its result.
         /// The <paramref name="parameters"/> argument is treated as: name1,value1,name2,value2,...
         /// </summary>
         protected virtual HttpResponseMessage ExecutePost(string request, HttpContent content, params string[]? parameters)
         {
             EnsureOpen();
 
-            var req = new HttpRequestMessage(HttpMethod.Post, QueryString(request, parameters));
-
-            req.Content = content;
+            var req = new HttpRequestMessage(HttpMethod.Post, QueryString(request, parameters))
+            {
+                Content = content
+            };
 
             return Execute(req);
         }
 
+        /// <summary>
+        /// <b>Internal:</b> Execute a POST request asynchronously with custom HttpContent.
+        /// The <paramref name="parameters"/> argument is treated as: name1,value1,name2,value2,...
+        /// </summary>
+        protected virtual async Task<HttpResponseMessage> ExecutePostAsync(string request, HttpContent content, params string[]? parameters)
+        {
+            EnsureOpen();
+
+            var req = new HttpRequestMessage(HttpMethod.Post, QueryString(request, parameters))
+            {
+                Content = content
+            };
+
+            var resp = await httpc.SendAsync(req).ConfigureAwait(false); // Async call

Review Comment:
   `ExecutePostAsync()` calls `httpc.SendAsync(req)` without specifying `HttpCompletionOption.ResponseHeadersRead`. If any POST endpoints return large payloads, this will buffer the full response in memory. Consider using `SendAsync(req, HttpCompletionOption.ResponseHeadersRead, ...)` (and ideally accepting/passing a `CancellationToken`) for consistency with the sync `Execute()` behavior.
   ```suggestion
               var resp = await httpc.SendAsync(req, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false); // Async call
   ```



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -244,20 +254,137 @@ private void DoUpdate()
             try
             {
                 if (notify && !disposed)
-                { // no use to notify if we are closed already
+                {
+                    // no use to notify if we are closed already
                     // LUCENENET specific - pass the copiedFiles as read only
                     handler.RevisionReady(session.Version, session.SourceFiles, Collections.AsReadOnly(copiedFiles), sourceDirectory);
                 }
             }
             finally
             {
                 IOUtils.Dispose(sourceDirectory.Values);
-                //TODO: Resharper Message, Expression is always true -> Verify and if so then we can remove the null check.
+
+                // LUCENENET specific: removed redundant check. Code above either ensures it is not null (by returning early) or an exception is thrown.
+                // if (session != null)
+                // {
+                    factory.CleanupSession(session.Id);
+                // }

Review Comment:
   In `DoUpdate()`, the final cleanup unconditionally calls `factory.CleanupSession(session.Id)`. If an exception occurs before `session` is assigned (or if `session` is null for any reason), this will throw a `NullReferenceException` in the `finally` block and can mask the original failure. Reinstate the `session != null` guard (or use `session?.Id` with an early return) before accessing `session.Id` here.
   ```suggestion
                   if (session != null)
                   {
                       factory.CleanupSession(session.Id);
                   }
   ```



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -244,20 +254,137 @@ private void DoUpdate()
             try
             {
                 if (notify && !disposed)
-                { // no use to notify if we are closed already
+                {
+                    // no use to notify if we are closed already
                     // LUCENENET specific - pass the copiedFiles as read only
                     handler.RevisionReady(session.Version, session.SourceFiles, Collections.AsReadOnly(copiedFiles), sourceDirectory);
                 }
             }
             finally
             {
                 IOUtils.Dispose(sourceDirectory.Values);
-                //TODO: Resharper Message, Expression is always true -> Verify and if so then we can remove the null check.
+
+                // LUCENENET specific: removed redundant check. Code above either ensures it is not null (by returning early) or an exception is thrown.
+                // if (session != null)
+                // {
+                    factory.CleanupSession(session.Id);
+                // }
+            }
+        }
+
+        /// <summary>
+        /// Performs the async update logic. When the replicator implements
+        /// <see cref="IAsyncReplicator"/>, native async methods are used;
+        /// otherwise, the synchronous <see cref="IReplicator"/> methods are
+        /// called and their results wrapped.
+        /// </summary>
+        private async Task DoUpdateAsync(CancellationToken cancellationToken)
+        {
+            SessionToken? session = null;
+            var sourceDirectory = new Dictionary<string, Directory>();
+            var copiedFiles = new Dictionary<string, IList<string>>();
+            bool notify = false;
+
+            try
+            {
+                string? version = handler.CurrentVersion;
+
+                session = asyncReplicator is not null
+                    ? await asyncReplicator.CheckForUpdateAsync(version, cancellationToken).ConfigureAwait(false)
+                    : replicator.CheckForUpdate(version);
+
+                WriteToInfoStream($"DoUpdateAsync(): handlerVersion={version} session={session}");
+
+                if (session is null)
+                    return;
+
+                IDictionary<string, IList<RevisionFile>> requiredFiles = RequiredFiles(session.SourceFiles);
+
+                foreach (var pair in requiredFiles)
+                {
+                    string source = pair.Key;
+                    Directory directory = factory.GetDirectory(session.Id!, source);
+
+                    sourceDirectory.Add(source, directory);
+                    IList<string> cpFiles = new JCG.List<string>();
+                    copiedFiles.Add(source, cpFiles);
+
+                    foreach (var file in pair.Value)
+                    {
+                        cancellationToken.ThrowIfCancellationRequested();
+
+                        Stream? input = null;
+                        IndexOutput? output = null;
+                        try
+                        {
+                            input = asyncReplicator is not null
+                                ? await asyncReplicator.ObtainFileAsync(session.Id!, source, file.FileName, cancellationToken).ConfigureAwait(false)
+                                : replicator.ObtainFile(session.Id!, source, file.FileName);
+
+                            output = directory.CreateOutput(file.FileName, IOContext.DEFAULT);
+
+                            int numBytes;
+                            byte[] buffer = new byte[16384];
+                            while ((numBytes = await input.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) > 0)
+                            {
+                                output.WriteBytes(buffer, 0, numBytes);
+                            }
+
+                            cpFiles.Add(file.FileName);
+                        }
+                        finally
+                        {
+                            IOUtils.Dispose(input, output);
+                        }
+                    }
+                }
+
+                notify = true;
+            }
+            finally
+            {
                 if (session != null)
                 {
-                    factory.CleanupSession(session.Id);
+                    try
+                    {
+                        if (asyncReplicator is not null)
+                            await asyncReplicator.ReleaseAsync(session.Id, cancellationToken).ConfigureAwait(false);

Review Comment:
   `DoUpdateAsync()` passes the caller’s `cancellationToken` into `ReleaseAsync()`. If cancellation is requested during file copy, the token will typically already be canceled by the time cleanup runs, which can prevent releasing the server-side session (and may mask the original exception). Cleanup/release should generally run even when the operation is cancelled; consider using `CancellationToken.None` (or a separate non-cancelable token) for `ReleaseAsync()` in the `finally` block.
   ```suggestion
                               await asyncReplicator.ReleaseAsync(session.Id, CancellationToken.None).ConfigureAwait(false);
   ```



##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -430,7 +557,125 @@ public virtual bool IsUpdateThreadAlive
                 UninterruptableMonitor.Enter(syncLock);
                 try
                 {
-                    return updateThread != null && updateThread.IsAlive;
+                    return updateThread is not null && updateThread.IsAlive;
+                }
+                finally
+                {
+                    UninterruptableMonitor.Exit(syncLock);
+                }
+            }
+        }
+
+        /// <summary>
+        /// Start a periodic async update loop with the specified interval in milliseconds.
+        /// </summary>
+        /// <param name="intervalInMilliseconds">Interval between updates.</param>
+        public virtual void StartAsyncUpdateLoop(long intervalInMilliseconds)
+        {
+            UninterruptableMonitor.Enter(syncLock);
+            try
+            {
+                EnsureOpen();
+
+                if (asyncUpdateTask != null && !asyncUpdateTask.IsCompleted)
+                    throw IllegalStateException.Create("Async update loop is already running. Stop it first.");
+
+                asyncUpdateCts = new CancellationTokenSource();
+                CancellationToken ct = asyncUpdateCts.Token;
+
+                asyncUpdateTask = Task.Run(async () =>
+                {
+                    while (!ct.IsCancellationRequested)
+                    {
+                        try
+                        {
+                            await updateLock.WaitAsync(ct).ConfigureAwait(false);
+                            try
+                            {
+                                await DoUpdateAsync(ct).ConfigureAwait(false);
+                            }
+                            finally
+                            {
+                                updateLock.Release();
+                            }
+                        }
+                        catch (TaskCanceledException)
+                        {
+                            // Cancellation requested, exit loop gracefully
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            HandleUpdateException(ex);
+                        }
+
+                        try
+                        {
+                            await Task.Delay(TimeSpan.FromMilliseconds(intervalInMilliseconds), ct).ConfigureAwait(false);
+                        }
+                        catch (TaskCanceledException)
+                        {
+                            break; // exit loop if cancelled during delay
+                        }
+                    }
+                }, ct);
+            }
+            finally
+            {
+                UninterruptableMonitor.Exit(syncLock);
+            }
+        }
+
+        /// <summary>
+        /// Stops the async update loop, if it is running.
+        /// </summary>
+        public virtual async Task StopAsyncUpdateLoop()
+        {
+            UninterruptableMonitor.Enter(syncLock);
+            try
+            {
+                if (asyncUpdateCts != null)
+                {
+#if FEATURE_CANCELLATIONTOKENSOURCE_CANCELASYNC
+                    await asyncUpdateCts.CancelAsync().ConfigureAwait(false);
+#else
+                    asyncUpdateCts.Cancel();
+#endif
+                    try
+                    {
+                        if (asyncUpdateTask != null)
+                        {
+                            await asyncUpdateTask.ConfigureAwait(false);
+                        }
+                    }
+                    catch (TaskCanceledException)
+                    {
+                    }
+                    finally
+                    {
+                        asyncUpdateTask = null;
+                        asyncUpdateCts.Dispose();
+                        asyncUpdateCts = null;
+                    }
+                }
+            }
+            finally
+            {
+                UninterruptableMonitor.Exit(syncLock);
+            }

Review Comment:
   `StopAsyncUpdateLoop()` awaits (`CancelAsync()` / `asyncUpdateTask`) while holding `syncLock`. Awaiting under a monitor lock can lead to deadlocks or prolonged blocking if any continuation/override code tries to acquire the same lock (directly or indirectly). Consider capturing `asyncUpdateCts`/`asyncUpdateTask` under the lock, releasing the lock, then performing the awaited operations, and finally reacquiring the lock to clear fields.
   ```suggestion
               CancellationTokenSource? localAsyncUpdateCts;
               Task? localAsyncUpdateTask;
   
               UninterruptableMonitor.Enter(syncLock);
               try
               {
                   localAsyncUpdateCts = asyncUpdateCts;
                   localAsyncUpdateTask = asyncUpdateTask;
               }
               finally
               {
                   UninterruptableMonitor.Exit(syncLock);
               }
   
               if (localAsyncUpdateCts != null)
               {
   #if FEATURE_CANCELLATIONTOKENSOURCE_CANCELASYNC
                   await localAsyncUpdateCts.CancelAsync().ConfigureAwait(false);
   #else
                   localAsyncUpdateCts.Cancel();
   #endif
                   try
                   {
                       if (localAsyncUpdateTask != null)
                       {
                           await localAsyncUpdateTask.ConfigureAwait(false);
                       }
                   }
                   catch (TaskCanceledException)
                   {
                   }
                   finally
                   {
                       localAsyncUpdateCts.Dispose();
   
                       UninterruptableMonitor.Enter(syncLock);
                       try
                       {
                           if (ReferenceEquals(asyncUpdateTask, localAsyncUpdateTask))
                           {
                               asyncUpdateTask = null;
                           }
   
                           if (ReferenceEquals(asyncUpdateCts, localAsyncUpdateCts))
                           {
                               asyncUpdateCts = null;
                           }
                       }
                       finally
                       {
                           UninterruptableMonitor.Exit(syncLock);
                       }
                   }
               }
   ```



##########
src/Lucene.Net.Replicator/Http/HttpClientBase.cs:
##########
@@ -200,6 +222,36 @@ protected virtual HttpResponseMessage ExecuteGet(string request, params string[]
             return Execute(req);
         }
 
+        /// <summary>
+        /// Execute a GET request asynchronously with an array of parameters.
+        /// </summary>
+        protected Task<HttpResponseMessage> ExecuteGetAsync(string action, string[]? parameters, CancellationToken cancellationToken)
+        {
+            EnsureOpen();
+            var url = QueryString(action, parameters);
+            return httpc.GetAsync(url, cancellationToken);
+        }

Review Comment:
   `ExecuteGetAsync(...)` uses `HttpClient.GetAsync(url, cancellationToken)` which defaults to `HttpCompletionOption.ResponseContentRead` (buffers the entire response). For replication file downloads this can cause large memory usage and defeats streaming. Use `SendAsync(..., HttpCompletionOption.ResponseHeadersRead, cancellationToken)` (similar to the sync `Execute()` path) so callers can stream the content without buffering.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add client-side async API for replication (IAsyncReplicator, ReplicationClient) [lucenenet]

Posted by "Shazwazza (via GitHub)" <gi...@apache.org>.
Shazwazza commented on code in PR #1182:
URL: https://github.com/apache/lucenenet/pull/1182#discussion_r3126981920


##########
src/Lucene.Net.Replicator/ReplicationClient.cs:
##########
@@ -151,6 +155,20 @@ public ReplicationClient(IReplicator replicator, IReplicationHandler handler, IS
             this.factory = factory;
         }
 
+        /// <summary>
+        /// Constructor for async replicators.
+        /// </summary>
+        /// <param name="asyncReplicator"></param>
+        /// <param name="handler"></param>
+        /// <param name="factory"></param>
+        /// <exception cref="ArgumentNullException"></exception>
+        public ReplicationClient(IAsyncReplicator asyncReplicator, IReplicationHandler handler, ISourceDirectoryFactory factory)

Review Comment:
   @paulirwin did you explore using the factory methods mentioned above? i.e. ReplicationClient.CreateAsyncClient(...)
   ReplicationClient.CreateClient(...)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org