[GitHub] [lucene-solr] mikemccand commented on a change in pull request #895: LUCENE-8985: SynonymGraphFilter cannot handle input stream with tokens filtered.

[GitBox <gi...@apache.org>]

mikemccand commented on a change in pull request #895: LUCENE-8985: SynonymGraphFilter cannot handle input stream with tokens filtered.
URL: https://github.com/apache/lucene-solr/pull/895#discussion_r356637259
 
 

 ##########
 File path: lucene/analysis/common/src/java/org/apache/lucene/analysis/synonym/SynonymGraphFilter.java
 ##########
 @@ -178,201 +157,136 @@ public SynonymGraphFilter(TokenStream input, SynonymMap synonyms, boolean ignore
     this.fstReader = fst.getBytesReader();
     scratchArc = new FST.Arc<>();
     this.ignoreCase = ignoreCase;
+    this.tokenQueue = new ArrayDeque<>();
+    this.lookahead = new ArrayDeque<>();
   }
 
   @Override
   public boolean incrementToken() throws IOException {
-    //System.out.println("\nS: incrToken lastNodeOut=" + lastNodeOut + " nextNodeOut=" + nextNodeOut);
-
-    assert lastNodeOut <= nextNodeOut;
-      
-    if (outputBuffer.isEmpty() == false) {
-      // We still have pending outputs from a prior synonym match:
-      releaseBufferedToken();
-      //System.out.println("  syn: ret buffered=" + this);
-      assert liveToken == false;
-      return true;
-    }
-
-    // Try to parse a new synonym match at the current token:
-
-    if (parse()) {
-      // A new match was found:
-      releaseBufferedToken();
-      //System.out.println("  syn: after parse, ret buffered=" + this);
-      assert liveToken == false;
-      return true;
-    }
-
-    if (lookaheadNextRead == lookaheadNextWrite) {
-
-      // Fast path: parse pulled one token, but it didn't match
-      // the start for any synonym, so we now return it "live" w/o having
-      // cloned all of its atts:
-      if (finished) {
-        //System.out.println("  syn: ret END");
+      if (!tokenQueue.isEmpty()){
 
 Review comment:
   Can you use `== false` instead, for better future readability?  And also add a space before `{`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org