[GitHub] [lucene] donnerpeter commented on a diff in pull request #11893: hunspell: allow for faster dictionary iteration during 'suggest' by using more memory (opt-in)

[GitBox <gi...@apache.org>]

donnerpeter commented on code in PR #11893:
URL: https://github.com/apache/lucene/pull/11893#discussion_r1013968050


##########
lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Suggester.java:
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.hunspell;
+
+import static org.apache.lucene.analysis.hunspell.Dictionary.FLAG_UNSET;
+import static org.apache.lucene.analysis.hunspell.TimeoutPolicy.NO_TIMEOUT;
+import static org.apache.lucene.analysis.hunspell.WordContext.COMPOUND_BEGIN;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.lucene.util.CharsRef;
+
+/**
+ * A generator for misspelled word corrections based on Hunspell flags. The suggestions are searched
+ * for in two main ways:
+ *
+ * <ol>
+ *   <li>Modification: trying to insert/remove/delete/swap parts of the word to get something
+ *       acceptable. The performance of this part depends heavily on the contents of TRY, MAP, REP,
+ *       KEY directives in the .aff file.
+ *   <li>Enumeration: if the modification hasn't produced "good enough" suggestions, the whole
+ *       dictionary is scanned and simple affixes are added onto the entries to check if that
+ *       produces anything similar to the given misspelled word. This depends on the dictionary size
+ *       and the affix count, and it can take noticeable amount of time. To speed this up, {@link
+ *       #withSuggestibleEntryCache()} can be used.
+ * </ol>
+ */
+public class Suggester {
+  private final Dictionary dictionary;
+  private final SuggestibleEntryCache suggestibleCache;
+
+  public Suggester(Dictionary dictionary) {
+    this(dictionary, null);
+  }
+
+  private Suggester(Dictionary dictionary, SuggestibleEntryCache suggestibleCache) {
+    this.dictionary = dictionary;
+    this.suggestibleCache = suggestibleCache;
+  }
+
+  /**
+   * Returns a copy of this suggester instance with better "Enumeration" phase performance (see
+   * {@link Suggester} documentation), but using more memory. With this option, the dictionary
+   * entries are stored as fast-to-iterate plain words instead of highly compressed prefix trees.
+   */
+  public Suggester withSuggestibleEntryCache() {

Review Comment:
   Thank you!
   
   I've tried to avoid making the cache also a publicly-accessible API because I don't think that clients need such detail. The cache's lifecycle can be longer than the one of the `checkCanceled` runnable (which is now bound to `Hunspell` unfortunately, but it turned out that we need to create it afresh on each `spell`/`suggest` call), not necessarily than the suggester's lifecycle. So far I envision clients creating a single suggester for a dictionary, customizing it in the way they need, and keeping it in the memory.
   
   If the cache remains private API, renaming it to `EntryCache` seems fine to me. If it should be made public, I'd leave something about suggestion there. But `SuggestionCache` means that suggestions themselves are cached, while here I cache some intermediate results.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org