You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by noblepaul <gi...@git.apache.org> on 2014/11/20 14:57:45 UTC

[GitHub] lucene-solr pull request: Create sparklr

GitHub user noblepaul opened a pull request:

    https://github.com/apache/lucene-solr/pull/105

    Create sparklr

    Testing out spark integration

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/noblepaul/lucene-solr patch-1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/lucene-solr/pull/105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #105
    
----
commit cfb2e8f57a78ded226b6cffd46bc88feff08018a
Author: noblepaul <no...@gmail.com>
Date:   2014-11-20T13:54:38Z

    Create sparklr
    
    Testing out spark integration

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85613814
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/FieldOffsetStrategy.java ---
    @@ -65,58 +65,88 @@ public String getField() {
        */
       public abstract List<OffsetsEnum> getOffsetsEnums(IndexReader reader, int docId, String content) throws IOException;
     
    -  protected List<OffsetsEnum> createOffsetsEnums(LeafReader leafReader, int doc, TokenStream tokenStream) throws IOException {
    -    List<OffsetsEnum> offsetsEnums = createOffsetsEnumsFromReader(leafReader, doc);
    -    if (automata.length > 0) {
    -      offsetsEnums.add(createOffsetsEnumFromTokenStream(doc, tokenStream));
    +  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader leafReader, int doc) throws IOException {
    +    final Terms termsIndex = leafReader.terms(field);
    +    if (termsIndex == null) {
    +      return Collections.emptyList();
         }
    -    return offsetsEnums;
    -  }
     
    -  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader atomicReader, int doc) throws IOException {
         // For strict positions, get a Map of term to Spans:
         //    note: ScriptPhraseHelper.NONE does the right thing for these method calls
         final Map<BytesRef, Spans> strictPhrasesTermToSpans =
    -        strictPhrases.getTermToSpans(atomicReader, doc);
    +        phraseHelper.getTermToSpans(leafReader, doc);
         // Usually simply wraps terms in a List; but if willRewrite() then can be expanded
         final List<BytesRef> sourceTerms =
    -        strictPhrases.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
    +        phraseHelper.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
     
    -    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + 1);
    +    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + automata.length);
     
    -    Terms termsIndex = atomicReader == null || sourceTerms.isEmpty() ? null : atomicReader.terms(field);
    -    if (termsIndex != null) {
    +    // Handle sourceTerms:
    +    if (!sourceTerms.isEmpty()) {
           TermsEnum termsEnum = termsIndex.iterator();//does not return null
           for (BytesRef term : sourceTerms) {
    -        if (!termsEnum.seekExact(term)) {
    -          continue; // term not found
    -        }
    -        PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    -        if (postingsEnum == null) {
    -          // no offsets or positions available
    -          throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    -        }
    -        if (doc != postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    -          continue;
    +        if (termsEnum.seekExact(term)) {
    +          PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    +
    +          if (postingsEnum == null) {
    +            // no offsets or positions available
    +            throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    +          }
    +
    +          if (doc == postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    +            postingsEnum = phraseHelper.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    +            if (postingsEnum != null) {
    +              offsetsEnums.add(new OffsetsEnum(term, postingsEnum));
    +            }
    +          }
             }
    -        postingsEnum = strictPhrases.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    -        if (postingsEnum == null) {
    -          continue;// completely filtered out
    +      }
    +    }
    +
    +    // Handle automata
    +    if (automata.length > 0) {
    +      offsetsEnums.addAll(createAutomataOffsetsFromTerms(termsIndex, doc));
    +    }
    +
    +    return offsetsEnums;
    +  }
    +
    +  protected List<OffsetsEnum> createAutomataOffsetsFromTerms(Terms termsIndex, int doc) throws IOException {
    +    Map<CharacterRunAutomaton, List<PostingsEnum>> automataPostings = new IdentityHashMap<>(automata.length);
    --- End diff --
    
    How about List<List<PostingsEnum>> that should give better locality without lose of type safety?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85607859
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/UnifiedHighlighter.java ---
    @@ -116,6 +116,8 @@
     
       private boolean defaultHighlightPhrasesStrictly = true; // AKA "accuracy" or "query debugging"
     
    +  private boolean defaultPassageRelevancyOverSpeed = true; //Prefer using a memory index
    --- End diff --
    
    Suggest the comment be: For analysis, prefer MemoryIndex approach


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Nevermind on `MultiPostingsEnum`.  MPE assumes the PostingsEnum are for a disjoint set of documents -- i.e. each PostingsEnum matches documents disjoint from the others.  This makes sense given it's use for a single view over multiple segments.  In our case we want what might be called a MultiPositionsPostingsEnum or MultiOffsetsPostingsEnum or "Composite" instead of "Multi" (synonymous, really).  We merge the positions (yet we don't care about the actual position #s; it's the other stuff we expose per-position).  BTW my code you merged had a "nocommit" to consider changing the class name.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Hmm, I know we're already knowingly breaking the PostingsEnum contract, but rather than throwing a IllegaStateException, maybe we could still return positions, they just happen to be unordered, that way when the final position is exhausted we return -1 as the sentinel value still?  Otherwise, I'm good with this, I'll merge in.  Btw, we've seen other needs for something like a CompositePostingsEnum that abstracts over a set of terms, but since this is still internal, dropping the house-keeping will also make this code do less. We can always try to cross that bridge later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85602210
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/AnalysisOffsetStrategy.java ---
    @@ -17,174 +17,28 @@
     package org.apache.lucene.search.uhighlight;
     
     import java.io.IOException;
    -import java.util.ArrayList;
    -import java.util.Arrays;
    -import java.util.Collections;
    -import java.util.List;
     
     import org.apache.lucene.analysis.Analyzer;
    -import org.apache.lucene.analysis.FilteringTokenFilter;
     import org.apache.lucene.analysis.TokenStream;
    -import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
    -import org.apache.lucene.index.IndexReader;
    -import org.apache.lucene.index.LeafReader;
    -import org.apache.lucene.index.Terms;
    -import org.apache.lucene.index.memory.MemoryIndex;
    -import org.apache.lucene.search.spans.SpanQuery;
     import org.apache.lucene.util.BytesRef;
    -import org.apache.lucene.util.automaton.Automata;
     import org.apache.lucene.util.automaton.CharacterRunAutomaton;
     
    +public abstract class AnalysisOffsetStrategy extends FieldOffsetStrategy {
    --- End diff --
    
    All public classes need a javadoc comment.  Remember lucene.internal for this one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Ah, missed that.  Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Check it out: https://github.com/dsmiley/lucene-solr/commit/50e2ea89c7eb15c863aa6e04e14fd32085ee85bd
    Remember this is a package-private class internal to the UnifiedHighlighter.  Not completely implementing an interface if the caller won't care is okay.
    
    RE MultiTermHighlighting: I think it's fine as-is.  The UH class is very long to adopt this -- too long as it is IMO.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85619786
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/FieldOffsetStrategy.java ---
    @@ -65,58 +65,88 @@ public String getField() {
        */
       public abstract List<OffsetsEnum> getOffsetsEnums(IndexReader reader, int docId, String content) throws IOException;
     
    -  protected List<OffsetsEnum> createOffsetsEnums(LeafReader leafReader, int doc, TokenStream tokenStream) throws IOException {
    -    List<OffsetsEnum> offsetsEnums = createOffsetsEnumsFromReader(leafReader, doc);
    -    if (automata.length > 0) {
    -      offsetsEnums.add(createOffsetsEnumFromTokenStream(doc, tokenStream));
    +  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader leafReader, int doc) throws IOException {
    +    final Terms termsIndex = leafReader.terms(field);
    +    if (termsIndex == null) {
    +      return Collections.emptyList();
         }
    -    return offsetsEnums;
    -  }
     
    -  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader atomicReader, int doc) throws IOException {
         // For strict positions, get a Map of term to Spans:
         //    note: ScriptPhraseHelper.NONE does the right thing for these method calls
         final Map<BytesRef, Spans> strictPhrasesTermToSpans =
    -        strictPhrases.getTermToSpans(atomicReader, doc);
    +        phraseHelper.getTermToSpans(leafReader, doc);
         // Usually simply wraps terms in a List; but if willRewrite() then can be expanded
         final List<BytesRef> sourceTerms =
    -        strictPhrases.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
    +        phraseHelper.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
     
    -    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + 1);
    +    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + automata.length);
     
    -    Terms termsIndex = atomicReader == null || sourceTerms.isEmpty() ? null : atomicReader.terms(field);
    -    if (termsIndex != null) {
    +    // Handle sourceTerms:
    +    if (!sourceTerms.isEmpty()) {
           TermsEnum termsEnum = termsIndex.iterator();//does not return null
           for (BytesRef term : sourceTerms) {
    -        if (!termsEnum.seekExact(term)) {
    -          continue; // term not found
    -        }
    -        PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    -        if (postingsEnum == null) {
    -          // no offsets or positions available
    -          throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    -        }
    -        if (doc != postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    -          continue;
    +        if (termsEnum.seekExact(term)) {
    +          PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    +
    +          if (postingsEnum == null) {
    +            // no offsets or positions available
    +            throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    +          }
    +
    +          if (doc == postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    +            postingsEnum = phraseHelper.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    +            if (postingsEnum != null) {
    +              offsetsEnums.add(new OffsetsEnum(term, postingsEnum));
    +            }
    +          }
             }
    -        postingsEnum = strictPhrases.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    -        if (postingsEnum == null) {
    -          continue;// completely filtered out
    +      }
    +    }
    +
    +    // Handle automata
    +    if (automata.length > 0) {
    +      offsetsEnums.addAll(createAutomataOffsetsFromTerms(termsIndex, doc));
    +    }
    +
    +    return offsetsEnums;
    +  }
    +
    +  protected List<OffsetsEnum> createAutomataOffsetsFromTerms(Terms termsIndex, int doc) throws IOException {
    +    Map<CharacterRunAutomaton, List<PostingsEnum>> automataPostings = new IdentityHashMap<>(automata.length);
    --- End diff --
    
    Pushed that for now to see what you think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85616651
  
    --- Diff: lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/visibility/TestUnifiedHighlighterExtensibility.java ---
    @@ -79,7 +90,7 @@ public void testFieldOffsetStrategyExtensibility() {
       @Test
       public void testUnifiedHighlighterExtensibility() {
         final int maxLength = 1000;
    -    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(random())){
    +    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(new Random())){
    --- End diff --
    
    I recall now why I changed it.  I started getting this error and figured it was a change elsewhere: java.lang.IllegalStateException: No context information for thread: Thread[id=1, name=main, state=RUNNABLE, group=main]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @AfterClass boundary (for example, static test class initializers are not permitted to access random contexts).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85612011
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/AnalysisOffsetStrategy.java ---
    @@ -17,174 +17,28 @@
     package org.apache.lucene.search.uhighlight;
     
     import java.io.IOException;
    -import java.util.ArrayList;
    -import java.util.Arrays;
    -import java.util.Collections;
    -import java.util.List;
     
     import org.apache.lucene.analysis.Analyzer;
    -import org.apache.lucene.analysis.FilteringTokenFilter;
     import org.apache.lucene.analysis.TokenStream;
    -import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
    -import org.apache.lucene.index.IndexReader;
    -import org.apache.lucene.index.LeafReader;
    -import org.apache.lucene.index.Terms;
    -import org.apache.lucene.index.memory.MemoryIndex;
    -import org.apache.lucene.search.spans.SpanQuery;
     import org.apache.lucene.util.BytesRef;
    -import org.apache.lucene.util.automaton.Automata;
     import org.apache.lucene.util.automaton.CharacterRunAutomaton;
     
    +public abstract class AnalysisOffsetStrategy extends FieldOffsetStrategy {
    --- End diff --
    
    Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85618489
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    --- End diff --
    
    Hmm, did you mean to calculate freq dynamically in the method and just use a remainingPositions count as walking over the postings? Instead of re-computing it's cached in the constructor, but if we modified it, the freq count would change as the postings were iterated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 closed the pull request at:

    https://github.com/apache/lucene-solr/pull/105


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85603003
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    +    private int position;
    +    private int nextPosition;
    +    private int positionInc = 1;
    +
    +    private int startOffset;
    --- End diff --
    
    Don't need these.  They can be fetched on-demand from the head of the queue, easily & cheaply enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Fair enough on both ends.  And yes, we have seen a potential use case for a unified view over terms.  I'll take a look at MultiPostingsEnum and see if we can use that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85603812
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    +    private int position;
    +    private int nextPosition;
    +    private int positionInc = 1;
    +
    +    private int startOffset;
    +    private int endOffset;
    +
    +    BoundsCheckingPostingsEnum(PostingsEnum postingsEnum) throws IOException {
    +      this.postingsEnum = postingsEnum;
    +      this.freq = postingsEnum.freq();
    +      nextPosition = postingsEnum.nextPosition();
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +    }
    +
    +    private boolean hasMorePositions() throws IOException {
    +      return positionInc < freq;
    +    }
    +
    +    /**
    +     * Returns the next position of the underlying postings enum unless
    +     * it cannot iterate further and returns NO_MORE_POSITIONS;
    +     * @return
    +     * @throws IOException
    +     */
    +    private int nextPosition() throws IOException {
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +      if (hasMorePositions()) {
    +        positionInc++;
    +        nextPosition = postingsEnum.nextPosition();
    +      } else {
    +        nextPosition = NO_MORE_POSITIONS;
    +      }
    +      return position;
    +    }
    +
    +  }
    +
    +  CompositePostingsEnum(BytesRef term, List<PostingsEnum> postingsEnums) throws IOException {
    +    this.term = term;
    +    queue = new PriorityQueue<BoundsCheckingPostingsEnum>(postingsEnums.size()) {
    +      @Override
    +      protected boolean lessThan(BoundsCheckingPostingsEnum a, BoundsCheckingPostingsEnum b) {
    +        return a.position < b.position;
    --- End diff --
    
    In the event the positions are equal (e.g. two terms a the same position in which the wildcard matches both), we might want to fall-back on startOffset then endOffset?  Or maybe simply ignore position altogether and just do offsets, so then you needn't even track the position?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Merged your commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85608645
  
    --- Diff: lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/visibility/TestUnifiedHighlighterExtensibility.java ---
    @@ -79,7 +90,7 @@ public void testFieldOffsetStrategyExtensibility() {
       @Test
       public void testUnifiedHighlighterExtensibility() {
         final int maxLength = 1000;
    -    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(random())){
    +    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(new Random())){
    --- End diff --
    
    ? why not `random()` ?  This will likely fail precommit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85606885
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/Passage.java ---
    @@ -40,7 +40,7 @@
         BytesRef matchTerms[] = new BytesRef[8];
         int numMatches = 0;
     
    -    void addMatch(int startOffset, int endOffset, BytesRef term) {
    +    public void addMatch(int startOffset, int endOffset, BytesRef term) {
    --- End diff --
    
    Excellent; now it's possible for someone to override `FieldHighlighter.highlightOffsetsEnums()` to make Passages.  But do add @lucene.experimental.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85613130
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/FieldOffsetStrategy.java ---
    @@ -65,58 +65,88 @@ public String getField() {
        */
       public abstract List<OffsetsEnum> getOffsetsEnums(IndexReader reader, int docId, String content) throws IOException;
     
    -  protected List<OffsetsEnum> createOffsetsEnums(LeafReader leafReader, int doc, TokenStream tokenStream) throws IOException {
    -    List<OffsetsEnum> offsetsEnums = createOffsetsEnumsFromReader(leafReader, doc);
    -    if (automata.length > 0) {
    -      offsetsEnums.add(createOffsetsEnumFromTokenStream(doc, tokenStream));
    +  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader leafReader, int doc) throws IOException {
    +    final Terms termsIndex = leafReader.terms(field);
    +    if (termsIndex == null) {
    +      return Collections.emptyList();
         }
    -    return offsetsEnums;
    -  }
     
    -  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader atomicReader, int doc) throws IOException {
         // For strict positions, get a Map of term to Spans:
         //    note: ScriptPhraseHelper.NONE does the right thing for these method calls
         final Map<BytesRef, Spans> strictPhrasesTermToSpans =
    -        strictPhrases.getTermToSpans(atomicReader, doc);
    +        phraseHelper.getTermToSpans(leafReader, doc);
         // Usually simply wraps terms in a List; but if willRewrite() then can be expanded
         final List<BytesRef> sourceTerms =
    -        strictPhrases.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
    +        phraseHelper.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
     
    -    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + 1);
    +    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + automata.length);
     
    -    Terms termsIndex = atomicReader == null || sourceTerms.isEmpty() ? null : atomicReader.terms(field);
    -    if (termsIndex != null) {
    +    // Handle sourceTerms:
    +    if (!sourceTerms.isEmpty()) {
           TermsEnum termsEnum = termsIndex.iterator();//does not return null
           for (BytesRef term : sourceTerms) {
    -        if (!termsEnum.seekExact(term)) {
    -          continue; // term not found
    -        }
    -        PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    -        if (postingsEnum == null) {
    -          // no offsets or positions available
    -          throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    -        }
    -        if (doc != postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    -          continue;
    +        if (termsEnum.seekExact(term)) {
    +          PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    +
    +          if (postingsEnum == null) {
    +            // no offsets or positions available
    +            throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    +          }
    +
    +          if (doc == postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    +            postingsEnum = phraseHelper.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    +            if (postingsEnum != null) {
    +              offsetsEnums.add(new OffsetsEnum(term, postingsEnum));
    +            }
    +          }
             }
    -        postingsEnum = strictPhrases.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    -        if (postingsEnum == null) {
    -          continue;// completely filtered out
    +      }
    +    }
    +
    +    // Handle automata
    +    if (automata.length > 0) {
    +      offsetsEnums.addAll(createAutomataOffsetsFromTerms(termsIndex, doc));
    +    }
    +
    +    return offsetsEnums;
    +  }
    +
    +  protected List<OffsetsEnum> createAutomataOffsetsFromTerms(Terms termsIndex, int doc) throws IOException {
    +    Map<CharacterRunAutomaton, List<PostingsEnum>> automataPostings = new IdentityHashMap<>(automata.length);
    --- End diff --
    
    One minor problem with that was that the code would not longer by type-safe because of the lack of generic arrays in java.  I wouldn't be able to do `List<PostingsEnum>[]` = new ArrayList<PostingsEnum>[automata.length];` but could do `List<PostingsEnum>[]` = new ArrayList[automata.length];` with unchecked casts.  Seem worth it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    I've pushed some more changes now.  Still taking a look at what we might be able to do further with CompositePostingsEnum


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85614277
  
    --- Diff: lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/visibility/TestUnifiedHighlighterExtensibility.java ---
    @@ -79,7 +90,7 @@ public void testFieldOffsetStrategyExtensibility() {
       @Test
       public void testUnifiedHighlighterExtensibility() {
         final int maxLength = 1000;
    -    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(random())){
    +    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(new Random())){
    --- End diff --
    
    fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85611673
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    +    private int position;
    +    private int nextPosition;
    +    private int positionInc = 1;
    +
    +    private int startOffset;
    --- End diff --
    
    I thought the same, but unfortunately there's no way to check the current position of the underlying PostingsEnum before calling nextPosition.  Since we call nextPosition the PostingsEnum has moved on before the highlighter calls startOffset or endOffset.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85604667
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    +    private int position;
    +    private int nextPosition;
    +    private int positionInc = 1;
    +
    +    private int startOffset;
    +    private int endOffset;
    +
    +    BoundsCheckingPostingsEnum(PostingsEnum postingsEnum) throws IOException {
    +      this.postingsEnum = postingsEnum;
    +      this.freq = postingsEnum.freq();
    +      nextPosition = postingsEnum.nextPosition();
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +    }
    +
    +    private boolean hasMorePositions() throws IOException {
    +      return positionInc < freq;
    +    }
    +
    +    /**
    +     * Returns the next position of the underlying postings enum unless
    +     * it cannot iterate further and returns NO_MORE_POSITIONS;
    +     * @return
    +     * @throws IOException
    +     */
    +    private int nextPosition() throws IOException {
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +      if (hasMorePositions()) {
    +        positionInc++;
    +        nextPosition = postingsEnum.nextPosition();
    +      } else {
    +        nextPosition = NO_MORE_POSITIONS;
    +      }
    +      return position;
    +    }
    +
    +  }
    +
    +  CompositePostingsEnum(BytesRef term, List<PostingsEnum> postingsEnums) throws IOException {
    +    this.term = term;
    +    queue = new PriorityQueue<BoundsCheckingPostingsEnum>(postingsEnums.size()) {
    +      @Override
    +      protected boolean lessThan(BoundsCheckingPostingsEnum a, BoundsCheckingPostingsEnum b) {
    +        return a.position < b.position;
    +      }
    +    };
    +
    +    int freqAdd = 0;
    +    for (PostingsEnum postingsEnum : postingsEnums) {
    +      queue.add(new BoundsCheckingPostingsEnum(postingsEnum));
    +      freqAdd += postingsEnum.freq();
    +    }
    +    freq = freqAdd;
    +  }
    +
    +  @Override
    +  public int freq() throws IOException {
    +    return freq;
    +  }
    +
    +  @Override
    +  public int nextPosition() throws IOException {
    +    int position = NO_MORE_POSITIONS;
    +    while (queue.size() >= 1) {
    +      queue.top().nextPosition();
    +      queue.updateTop(); //the new position may be behind another postingsEnum in the queue
    +      position = queue.top().position;
    +
    +      if (position == NO_MORE_POSITIONS) {
    +        queue.pop(); //this postingsEnum is consumed, let's get rid of it
    +      } else {
    +        break; //we got a new position
    +      }
    +
    +    }
    +    return position;
    +  }
    +
    +  @Override
    +  public int startOffset() throws IOException {
    +    return queue.top().startOffset;
    +  }
    +
    +  @Override
    +  public int endOffset() throws IOException {
    +    return queue.top().endOffset;
    +  }
    +
    +  @Override
    +  public BytesRef getPayload() throws IOException {
    +    //The UnifiedHighlighter depends on the payload for a wildcard
    +    //being the term representing it
    +    return term;
    +  }
    +
    +  @Override
    +  public int docID() {
    +    return queue.top().postingsEnum.docID();
    --- End diff --
    
    I think this one should be held in a field of CompositePostingsEnum initialized in constructor?  Not a big deal though, since I don't think the highlighter is going to call this, let alone sometime after positions are exhausted (which would trigger an NPE in this case).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    I don't think there's a way to avoid keeping the position state, unfortunately.  The reason is that we can move one of the postings enums to the next position, but then realize the next position for that term is behind the position for a different term (and postings enum) that also matches the wildcard.  Then we'll update the top and switch to the next postings enum (by offset now), but once it's exhausted or we switch back to the previous one from interleaving the position is lost.  :/  An alternative to avoid this would be to change PostingsEnum to allow fetching of the currentPosition, then nearly all the house keeping would go away.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    _(I wrote this as you made your last comments, but rather than delete it I'll comment any way)_
    
    The documentation for `PostingsEnum.nextPosition()` states that calling it more than `freq()` times is undefined.  Thus it's quite valid to throw an IllegalStateException.
    
    > Btw, we've seen other needs for something like a CompositePostingsEnum that abstracts over a set of terms, but since this is still internal, dropping the house-keeping will also make this code do less. 
    
    I don't think I quite get what you're saying.  By "other needs" do you mean Bloomberg internally?  If so, how would that relate this this one inside the UH?  Are you advocating a general purpose Multi-PosrtingsEnum?  On the latter... a highlighter wouldn't be where to introduce such a thing.  There is a `org.apache.lucene.index.MultiPostingsEnum` which I looked at while at the Lucene hackday code sprint as it got my curiosity.  Unfortunately, it doesn't seem quite general purpose enough for us to use -- it demands a MultiTermsEnum parent.  Perhaps that could be improved to be useful without demanding a MultiTermsEnum parent... but that seems like a separate issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85611978
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    +    private int position;
    +    private int nextPosition;
    +    private int positionInc = 1;
    +
    +    private int startOffset;
    +    private int endOffset;
    +
    +    BoundsCheckingPostingsEnum(PostingsEnum postingsEnum) throws IOException {
    +      this.postingsEnum = postingsEnum;
    +      this.freq = postingsEnum.freq();
    +      nextPosition = postingsEnum.nextPosition();
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +    }
    +
    +    private boolean hasMorePositions() throws IOException {
    +      return positionInc < freq;
    +    }
    +
    +    /**
    +     * Returns the next position of the underlying postings enum unless
    +     * it cannot iterate further and returns NO_MORE_POSITIONS;
    +     * @return
    +     * @throws IOException
    +     */
    +    private int nextPosition() throws IOException {
    +      position = nextPosition;
    +      startOffset = postingsEnum.startOffset();
    +      endOffset = postingsEnum.endOffset();
    +      if (hasMorePositions()) {
    +        positionInc++;
    +        nextPosition = postingsEnum.nextPosition();
    +      } else {
    +        nextPosition = NO_MORE_POSITIONS;
    +      }
    +      return position;
    +    }
    +
    +  }
    +
    +  CompositePostingsEnum(BytesRef term, List<PostingsEnum> postingsEnums) throws IOException {
    +    this.term = term;
    +    queue = new PriorityQueue<BoundsCheckingPostingsEnum>(postingsEnums.size()) {
    +      @Override
    +      protected boolean lessThan(BoundsCheckingPostingsEnum a, BoundsCheckingPostingsEnum b) {
    +        return a.position < b.position;
    +      }
    +    };
    +
    +    int freqAdd = 0;
    +    for (PostingsEnum postingsEnum : postingsEnums) {
    +      queue.add(new BoundsCheckingPostingsEnum(postingsEnum));
    +      freqAdd += postingsEnum.freq();
    +    }
    +    freq = freqAdd;
    +  }
    +
    +  @Override
    +  public int freq() throws IOException {
    +    return freq;
    +  }
    +
    +  @Override
    +  public int nextPosition() throws IOException {
    +    int position = NO_MORE_POSITIONS;
    +    while (queue.size() >= 1) {
    +      queue.top().nextPosition();
    +      queue.updateTop(); //the new position may be behind another postingsEnum in the queue
    +      position = queue.top().position;
    +
    +      if (position == NO_MORE_POSITIONS) {
    +        queue.pop(); //this postingsEnum is consumed, let's get rid of it
    +      } else {
    +        break; //we got a new position
    +      }
    +
    +    }
    +    return position;
    +  }
    +
    +  @Override
    +  public int startOffset() throws IOException {
    +    return queue.top().startOffset;
    +  }
    +
    +  @Override
    +  public int endOffset() throws IOException {
    +    return queue.top().endOffset;
    +  }
    +
    +  @Override
    +  public BytesRef getPayload() throws IOException {
    +    //The UnifiedHighlighter depends on the payload for a wildcard
    +    //being the term representing it
    +    return term;
    +  }
    +
    +  @Override
    +  public int docID() {
    +    return queue.top().postingsEnum.docID();
    --- End diff --
    
    K


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85606333
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/FieldOffsetStrategy.java ---
    @@ -65,58 +65,88 @@ public String getField() {
        */
       public abstract List<OffsetsEnum> getOffsetsEnums(IndexReader reader, int docId, String content) throws IOException;
     
    -  protected List<OffsetsEnum> createOffsetsEnums(LeafReader leafReader, int doc, TokenStream tokenStream) throws IOException {
    -    List<OffsetsEnum> offsetsEnums = createOffsetsEnumsFromReader(leafReader, doc);
    -    if (automata.length > 0) {
    -      offsetsEnums.add(createOffsetsEnumFromTokenStream(doc, tokenStream));
    +  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader leafReader, int doc) throws IOException {
    +    final Terms termsIndex = leafReader.terms(field);
    +    if (termsIndex == null) {
    +      return Collections.emptyList();
         }
    -    return offsetsEnums;
    -  }
     
    -  protected List<OffsetsEnum> createOffsetsEnumsFromReader(LeafReader atomicReader, int doc) throws IOException {
         // For strict positions, get a Map of term to Spans:
         //    note: ScriptPhraseHelper.NONE does the right thing for these method calls
         final Map<BytesRef, Spans> strictPhrasesTermToSpans =
    -        strictPhrases.getTermToSpans(atomicReader, doc);
    +        phraseHelper.getTermToSpans(leafReader, doc);
         // Usually simply wraps terms in a List; but if willRewrite() then can be expanded
         final List<BytesRef> sourceTerms =
    -        strictPhrases.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
    +        phraseHelper.expandTermsIfRewrite(terms, strictPhrasesTermToSpans);
     
    -    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + 1);
    +    final List<OffsetsEnum> offsetsEnums = new ArrayList<>(sourceTerms.size() + automata.length);
     
    -    Terms termsIndex = atomicReader == null || sourceTerms.isEmpty() ? null : atomicReader.terms(field);
    -    if (termsIndex != null) {
    +    // Handle sourceTerms:
    +    if (!sourceTerms.isEmpty()) {
           TermsEnum termsEnum = termsIndex.iterator();//does not return null
           for (BytesRef term : sourceTerms) {
    -        if (!termsEnum.seekExact(term)) {
    -          continue; // term not found
    -        }
    -        PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    -        if (postingsEnum == null) {
    -          // no offsets or positions available
    -          throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    -        }
    -        if (doc != postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    -          continue;
    +        if (termsEnum.seekExact(term)) {
    +          PostingsEnum postingsEnum = termsEnum.postings(null, PostingsEnum.OFFSETS);
    +
    +          if (postingsEnum == null) {
    +            // no offsets or positions available
    +            throw new IllegalArgumentException("field '" + field + "' was indexed without offsets, cannot highlight");
    +          }
    +
    +          if (doc == postingsEnum.advance(doc)) { // now it's positioned, although may be exhausted
    +            postingsEnum = phraseHelper.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    +            if (postingsEnum != null) {
    +              offsetsEnums.add(new OffsetsEnum(term, postingsEnum));
    +            }
    +          }
             }
    -        postingsEnum = strictPhrases.filterPostings(term, postingsEnum, strictPhrasesTermToSpans.get(term));
    -        if (postingsEnum == null) {
    -          continue;// completely filtered out
    +      }
    +    }
    +
    +    // Handle automata
    +    if (automata.length > 0) {
    +      offsetsEnums.addAll(createAutomataOffsetsFromTerms(termsIndex, doc));
    +    }
    +
    +    return offsetsEnums;
    +  }
    +
    +  protected List<OffsetsEnum> createAutomataOffsetsFromTerms(Terms termsIndex, int doc) throws IOException {
    +    Map<CharacterRunAutomaton, List<PostingsEnum>> automataPostings = new IdentityHashMap<>(automata.length);
    --- End diff --
    
    I suggest a parallel array to automata, so that later you can avoid a map lookup on each matching term.  Also, I suggest lazy-initializing the array later... perhaps some wildcards in a disjunction might never match.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Hmm, clever! But not sure I find it very clean though.  I feel like that can lead to trouble down the road if code ever expects the offsets to be ordered.  If we went that route we wouldn't even need the priority queue though.  Btw, I MultiTermHighlighting is nearly gone except for one method that is used in the UnifiedHighlighter and MemoryIndexOffsetStrategy for extracting automata from a query.  Any ideas on good ways to move it? Perhaps the UnifiedHighlighter should do all automata extraction and pass that in? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request: Create sparklr

Posted by noblepaul <gi...@git.apache.org>.
Github user noblepaul closed the pull request at:

    https://github.com/apache/lucene-solr/pull/105


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    I started playing with a bit and realized the same thing.  It looks straight-forward but it's deceptively more complicated.  Then it hit me -- lets not try to return the correct position at all!  A "normal" PostingsEnum should but this one is _only_ used for offsets.  So always return -1 -- we can get away with it for this internal use.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr issue #105: LUCENE-7526 Improvements to UnifiedHighlighter Offse...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on the issue:

    https://github.com/apache/lucene-solr/pull/105
  
    Forget it, you can't do that because the next position on one enum might be -1, but there's more enums left in the queue so the user of this class could inadvertently terminate early if looking for -1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85602404
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    --- End diff --
    
    A comment would be helpful to explain the scope/purpose.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by Timothy055 <gi...@git.apache.org>.
Github user Timothy055 commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85616894
  
    --- Diff: lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/visibility/TestUnifiedHighlighterExtensibility.java ---
    @@ -79,7 +90,7 @@ public void testFieldOffsetStrategyExtensibility() {
       @Test
       public void testUnifiedHighlighterExtensibility() {
         final int maxLength = 1000;
    -    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(random())){
    +    UnifiedHighlighter uh = new UnifiedHighlighter(null, new MockAnalyzer(new Random())){
    --- End diff --
    
    I'm a bit confused.  What I should I change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85607262
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/TokenStreamOffsetStrategy.java ---
    @@ -0,0 +1,60 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.Closeable;
    +import java.io.IOException;
    +import java.util.Collections;
    +import java.util.List;
    +
    +import org.apache.lucene.analysis.Analyzer;
    +import org.apache.lucene.analysis.TokenStream;
    +import org.apache.lucene.index.IndexReader;
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.automaton.CharacterRunAutomaton;
    +
    +public class TokenStreamOffsetStrategy extends AnalysisOffsetStrategy {
    +
    +  private static final BytesRef[] ZERO_LEN_BYTES_REF_ARRAY = new BytesRef[0];
    +
    +  public TokenStreamOffsetStrategy(String field, BytesRef[] terms, PhraseHelper phraseHelper, CharacterRunAutomaton[] automata, Analyzer indexAnalyzer) {
    +    super(field, terms, phraseHelper, automata, indexAnalyzer);
    +    this.automata = convertTermsToAutomata(terms, automata);
    +    this.terms = ZERO_LEN_BYTES_REF_ARRAY;
    +  }
    +
    +  @Override
    +  public List<OffsetsEnum> getOffsetsEnums(IndexReader reader, int docId, String content) throws IOException {
    +    TokenStream tokenStream = tokenStream(content);
    +    PostingsEnum mtqPostingsEnum = MultiTermHighlighting.getDocsEnum(tokenStream, automata);
    --- End diff --
    
    I think there's a case to be made in moving ` MultiTermHighlighting.getDocsEnum` into this class, to thus keep the TokenStream aspect more isolated?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[GitHub] lucene-solr pull request #105: LUCENE-7526 Improvements to UnifiedHighlighte...

Posted by dsmiley <gi...@git.apache.org>.
Github user dsmiley commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/105#discussion_r85602867
  
    --- Diff: lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/CompositePostingsEnum.java ---
    @@ -0,0 +1,165 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.lucene.search.uhighlight;
    +
    +import java.io.IOException;
    +import java.util.List;
    +
    +import org.apache.lucene.index.PostingsEnum;
    +import org.apache.lucene.util.BytesRef;
    +import org.apache.lucene.util.PriorityQueue;
    +
    +
    +final class CompositePostingsEnum extends PostingsEnum {
    +
    +  private static final int NO_MORE_POSITIONS = -2;
    +  private final BytesRef term;
    +  private final int freq;
    +  private final PriorityQueue<BoundsCheckingPostingsEnum> queue;
    +
    +
    +  /**
    +   * This class is used to ensure we don't over iterate the underlying
    +   * postings enum by keeping track of the position relative to the
    +   * frequency.
    +   * Ideally this would've been an implementation of a PostingsEnum
    +   * but it would have to delegate most methods and it seemed easier
    +   * to just wrap the tweaked method.
    +   */
    +  private static final class BoundsCheckingPostingsEnum {
    +
    +
    +    private final PostingsEnum postingsEnum;
    +    private final int freq;
    --- End diff --
    
    Instead of holding `freq` and `nextPosition`, why not just `remainingPositions`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org