You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "rnewson (via GitHub)" <gi...@apache.org> on 2024/06/12 21:34:43 UTC

[PR] Use our own logic for index management [couchdb]

rnewson opened a new pull request, #5088:
URL: https://github.com/apache/couchdb/pull/5088

   ## Overview
   
   Use our own logic for index management
       
   Caffeine complaines when eviction is slow (because we might commit the index).
   Build our own logic for opening and closing indexes which allows opening and
   closing to be very slow, while ensuring we don't close an index that's in use.
   
   ## Testing recommendations
   
   covered by tests
   
   ## Related Issues or Pull Requests
   
   N/A
   
   ## Checklist
   
   - [x] Code is written and works correctly
   - [x] Changes are covered by tests
   - [ ] Any new configurable parameters are documented in `rel/overlay/etc/default.ini`
   - [ ] Documentation changes were made in the `src/docs` folder
   - [ ] Documentation changes were backported (separated PR) to affected branches
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637957391


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   Yes, 
   
     Lock downgrading
   
     Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. 
   
   https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "ben-manes (via GitHub)" <gi...@apache.org>.
ben-manes commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2168313452

   Since the mapping is discarded asynchronously, I suppose `removeEldestEntry` could try to evict the same one multiple times during a slow unloading? In those cases then once that completes the cache would still exceed its capacity and be stuck there, since the call only happens once on insertion. I think you would need to prune the cache yourself after unloading to avoid a memory leak.
   
   If you need a quick-and-dirty concurrent cache, the simple and classic and earliest approaches are to use a Clock (aka SecondChance) or a random sampling based strategy. These are popular, easy to write and maintain, can have good hit rates, are lock-free on reads, and the longer lock hold time on a write is typically okay for small caches. The exact workloads where they are good and bad varies, so it can require algorithm tuning to maximize hit rates, whereas Caffeine does that automatically.
   
   It sounds like the key insight of your approach is to evict the entry from the eviction policy while asynchronously removing it from the data map. Caffeine works the other way by keeping the data map linearizable (as user-visible) and the policy eventually consistent. If you decided to keep using Caffeine then you could have it act as the policy but store the data in a another hash table and use the loader + listener for coordination. You can think of that as an L1 / L2 inclusive cache where L1 pulls from L2 and L2 invalidations are sent to L1. That's not uncommon for local + remote cache layers, but here both would be on-heap.
   
   However you go about it, it sounds like a fun but tricky little problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1638303079


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   I was thinking if `readLock().lock()` could fail (does it throw or get interrupted?) then the next statement `holder.lock.writeLock().unlock();` won't be run and it would be stuck with a write lock taken.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson merged PR #5088:
URL: https://github.com/apache/couchdb/pull/5088


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "ben-manes (via GitHub)" <gi...@apache.org>.
ben-manes commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2167111488

   did you intend to have `IndexEvictionListener` run atomically within the map operation that removes the entry? In your new logic you defer that work to be asynchronous, so I believe you could use `Caffeine.removalListener` with `cause.wasEvicted()` for the same effect. The naming is a little unobvious, unfortunately, and a characteristic of feature evolution and keeping familiarity with Guava Cache for straightforward migrations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "ben-manes (via GitHub)" <gi...@apache.org>.
ben-manes commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2176636360

   @rnewson did you review if there is a memory leak as I suspected? The scenario is,
   
   1. Insert N items
   2. Insert key N+1 to trigger an eviction for key K0 (removeEldestEntry => false)
   3. Unload starts
   4. Insert K entries (removeEldestEntry => false, still selects key=0)
   5. Unload finished
   6. Insert key X => what is evicted?
   
   I believe since `removeEldestEntry` returns false it will be stuck growing to higher thresholds based on the number of observed concurrent evictions. Since unloading is slow, over time the cache could greatly exceed your threshold. That excess won't be unloaded since only the eldest in-flight unload is observable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1638733884


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,33 +114,97 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            // CachedData pattern from ReentrantReadWriteLock javadoc
+            holder.lock.readLock().lock();
 
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
+            // Load if not already loaded.
+            if (holder.state == HolderState.NOT_LOADED) {
+                holder.lock.readLock().unlock();
+                holder.lock.writeLock().lock();
                 try {
-                    return indexFun.apply(index);
+                    if (holder.state == HolderState.NOT_LOADED) {
+                        holder.index = load(name);
+                        holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                commitFun(name, holder),
+                                commitIntervalSeconds,
+                                commitIntervalSeconds,
+                                TimeUnit.SECONDS);
+                        holder.state = HolderState.LOADED;
+                    }
+                    holder.lock.readLock().lock();
                 } finally {
-                    index.release();
+                    holder.lock.writeLock().unlock();
                 }
             }
+
+            try {
+                switch (holder.state) {
+                    case NOT_LOADED:
+                        throw new IllegalStateException();
+                    case UNLOADED:
+                        Thread.sleep(1000);
+                        continue retry;
+                    case LOADED:
+                        return indexFun.apply(holder.index);
+                }
+            } finally {
+                holder.lock.readLock().unlock();
+            }
+        }
+    }
+
+    public void unload(final String name, final boolean forceDelete) throws IOException, InterruptedException {
+        final IndexHolder holder;
+        synchronized (cache) {
+            holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+        }
+        holder.lock.writeLock().lock();
+        try {
+            switch (holder.state) {
+                case LOADED:
+                    if (forceDelete) {
+                        holder.index.setDeleteOnClose(true);
+                    }
+                    LOGGER.info("closing {}", name);
+                    try {
+                        close(name, holder);
+                    } catch (final IOException e) {
+                        LOGGER.error("I/O exception when evicting {}", name, e);
+                    }
+                    holder.state = HolderState.UNLOADED;
+                    holder.index = null;
+                    synchronized (cache) {
+                        cache.remove(name, holder);
+                    }
+                    break;
+                case NOT_LOADED:
+                case UNLOADED:
+                    if (forceDelete) {
+                        IOUtils.rm(indexRootPath(name));
+                    }

Review Comment:
   hm, perhaps. I am also pondering if we need three states. It is important not to block reading the cache while we load or unload (as these can take many seconds), hence I let you retrieve from the cache under just the cache lock, though it might return a Holder in NOT_LOADED or UNLOADED states. the first thread to acquire the write lock gets the chore of opening the index. Perhaps I really don't need a distinct UNLOADED state? It could go back to UN_LOADED, and then we'd try to load it again, but only if someone has acquired the Holder before it is removed from the cache map itself...  Hm, but no, because then we'd have to reinsert into the map. I think a boring lifecycle is better, NOT_LOADED -> LOADED -> UN_LOADED (and removed from the cache), and repeat. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1638371860


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   updated. agree that the read lock() should be before the finally for the reason stated. pattern now matches the javadoc example more closely.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637209860


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   If we hold the write lock we're allowed to take a read lock and now we hold both?
   
   I wonder if ` holder.lock.readLock().lock();` should be before the `} finally {` instead of after, in other words `finally` should only handle unlocking the write lock we took above in line 141 ` holder.lock.writeLock()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1638356958


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   ah, I see what you mean. will ponder. the proper spell for this in is those javadocs, will double check
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1638507209


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,33 +114,97 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            // CachedData pattern from ReentrantReadWriteLock javadoc
+            holder.lock.readLock().lock();
 
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
+            // Load if not already loaded.
+            if (holder.state == HolderState.NOT_LOADED) {
+                holder.lock.readLock().unlock();
+                holder.lock.writeLock().lock();
                 try {
-                    return indexFun.apply(index);
+                    if (holder.state == HolderState.NOT_LOADED) {
+                        holder.index = load(name);
+                        holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                commitFun(name, holder),
+                                commitIntervalSeconds,
+                                commitIntervalSeconds,
+                                TimeUnit.SECONDS);
+                        holder.state = HolderState.LOADED;
+                    }
+                    holder.lock.readLock().lock();
                 } finally {
-                    index.release();
+                    holder.lock.writeLock().unlock();
                 }
             }
+
+            try {
+                switch (holder.state) {
+                    case NOT_LOADED:
+                        throw new IllegalStateException();
+                    case UNLOADED:
+                        Thread.sleep(1000);
+                        continue retry;
+                    case LOADED:
+                        return indexFun.apply(holder.index);
+                }
+            } finally {
+                holder.lock.readLock().unlock();
+            }
+        }
+    }
+
+    public void unload(final String name, final boolean forceDelete) throws IOException, InterruptedException {
+        final IndexHolder holder;
+        synchronized (cache) {
+            holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+        }
+        holder.lock.writeLock().lock();
+        try {
+            switch (holder.state) {
+                case LOADED:
+                    if (forceDelete) {
+                        holder.index.setDeleteOnClose(true);
+                    }
+                    LOGGER.info("closing {}", name);
+                    try {
+                        close(name, holder);
+                    } catch (final IOException e) {
+                        LOGGER.error("I/O exception when evicting {}", name, e);
+                    }
+                    holder.state = HolderState.UNLOADED;
+                    holder.index = null;
+                    synchronized (cache) {
+                        cache.remove(name, holder);
+                    }
+                    break;
+                case NOT_LOADED:
+                case UNLOADED:
+                    if (forceDelete) {
+                        IOUtils.rm(indexRootPath(name));
+                    }

Review Comment:
   Are there any assertions we can make about the presence in cache in these NOT_LOADED/UNLOADED state? For instance, would it make sense to assert we're not in the cache at this point?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2176854320

   ah, oops. will check, and fix in subsequent PR. thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637957391


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   Yes, 
   ```
   Lock downgrading
   
   Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. 
   ```
   https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637957391


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   Yes, 
   
   > Lock downgrading
   > 
   > Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. 
   
   https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2167524989

   @ben-manes Hi, and thanks for jumping in (and for developing Caffeine). Yes, I need the eviction to happen atomically (or, at least, no client should try to use the item while it is unloading), and it can be very slow (potentially minutes). I concluded that I was using Caffeine inappropriately; caching stateful objects that have a mutex external to the JVM, a file-based lock, where loading _and_ evicting can be very slow.
   
   I need to ensure that any given cache entry is loaded and not unloading at the time of any interaction. I also can't load the same item twice (Lucene's write.lock is preventing that for very good reasons), which made the race between an eviction and a concurrent get() somewhat fun.
   
   As you've seen I'm just using `LinkedHashMap` as a way to trigger awareness that we're at capacity, and I perform the unloading later. Happy to just accept that we might go a little over capacity, the trade-off in code simplicity is worth it to me.
   
   Your suggestions are very helpful though, and I might be pulled back to Caffeine for this as I get more results from real world performance (I am not too thrilled about the synchronized(cache) bits, especially when reading the cache). What I like about this PR though is it pulls all of the concurrency logic into IndexManager directly, and there's not too much of it. Time will tell if it holds up, of course.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2176504144

   thanks Nick. I expect we'll find a corner case but at least we have all the logic in one place, and it's quite small. Time will tell, and we clearly have some other options to try from what Ben has contributed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637960107


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   the purpose here is to ensure we have a read lock or write lock at all times.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#discussion_r1637957391


##########
nouveau/src/main/java/org/apache/couchdb/nouveau/core/IndexManager.java:
##########
@@ -82,43 +114,104 @@ public interface IndexFunction<V, R> {
 
     private ObjectMapper objectMapper;
 
-    private MetricRegistry metricRegistry;
-
-    private Scheduler scheduler;
+    private ScheduledExecutorService schedulerExecutorService;
 
     private SearcherFactory searcherFactory;
 
-    private AsyncLoadingCache<String, Index> cache;
+    private LRUMap cache;
 
-    private StripedLock<String> lock;
+    private StripedLock<String> createLock;
 
     public <R> R with(final String name, final IndexFunction<Index, R> indexFun)
             throws IOException, InterruptedException {
+        retry:
         while (true) {
             if (!exists(name)) {
                 throw new WebApplicationException("Index does not exist", Status.NOT_FOUND);
             }
-
-            final CompletableFuture<Index> future = cache.get(name);
-            final Index index = future.join();
-
-            if (index.tryAcquire()) {
-                try {
-                    return indexFun.apply(index);
-                } finally {
-                    index.release();
+            final IndexHolder holder;
+            synchronized (cache) {
+                holder = cache.computeIfAbsent(name, (k) -> new IndexHolder());
+            }
+            holder.lock.readLock().lock();
+            try {
+                // Load if not already loaded.
+                if (holder.state == HolderState.NOT_LOADED) {
+                    holder.lock.readLock().unlock();
+                    holder.lock.writeLock().lock();
+                    try {
+                        if (holder.state == HolderState.NOT_LOADED) {
+                            holder.index = load(name);
+                            holder.commitFuture = this.schedulerExecutorService.scheduleWithFixedDelay(
+                                    commitFun(name, holder),
+                                    commitIntervalSeconds,
+                                    commitIntervalSeconds,
+                                    TimeUnit.SECONDS);
+                            holder.state = HolderState.LOADED;
+                        }
+                    } finally {
+                        holder.lock.readLock().lock();

Review Comment:
   Yes, ```
   Lock downgrading
   
   Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. 
   ```
   https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use our own logic for index management [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5088:
URL: https://github.com/apache/couchdb/pull/5088#issuecomment-2168331840

   hmm, maybe. concurrent attempts to cache.get() while we're unloading should all get the IndexHolder while it is either write-locked (currently unloading) or not locked but in the UN_LOADED state (in which event we retry, as the entry should be removed from the cache and the first thread that gets the lock on `cache` will create a new IndexHolder in NOT_LOADED state). However, it's exactly all these kinds of cases that I wanted to delegate to something else, like Caffeine, so I will be holding off merging this PR while I ponder it some more.
   
   You've given me a lot to think about, and, yes, it _is_ a surprisingly fun little problem to solve. You'd think on my third "full text for couchdb" codebase that I'd have a solid solution to it by now. alas...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org