Highlighting: Analyzing query using analyzers for hl.fl fields

[Chris Harris <ry...@gmail.com>]

Currently during highlighting, the query string is analyzed by the
analyzer returned by IndexSchema.getQueryAnalyzer(). (If you step through
the code, you'll see that the Query object representing the
analyzed-and-parsed query string is generated before SolrHighlighter's
key doHighlighting() method gets called.)

Two things to emphasized here:

 * Query analysis takes place independent of which field is being
highlighted. (In other words, the query analyzer used does not vary
depending on which hl.fl is currently under consideration.)
 * Under the hood, this analyzer delegates to a separate sub-Analyzer for each
field referenced in the query itself. (For example, if you have the
query "body_text:smith AND num:5", then "smith" will perhaps be analyzed
using an analyzer with stopword analysis, stemming, etc., while "5" will
be analyzed with something simpler, more appropriate for a numeric-only
field.)

Or, to summarize: Query analysis during highlighting is a function of
the fields being *searched*, and *not* a function of the fields being
*highlighted*.

It seems to me that this behavior might be backwards. That is, what
we'd really want is for query analysis during highlighting to be a
function of the fields being highlighted (i.e. the hl.fl params), and
*not* of the fields being mentioned in the query.

Let me try to sketch the use case that leads me to think this:

I have an index with two fields:

  body (default field; word bigram analyzer)
  kwic (text is copyTo'd here from the body field; non-bigram analyzer)

(By "word bigram analyzer" I mean one that might analyze the input
"once upon a time" into the token stream "once", "once upon", "upon",
"upon a", "a", "a time".)

Let's say I want to search for

  "audit trail"

(with quotes), and get use hl.fl=kwic. If I use the current highlight
mechanism, then it will be the *body* field's bigram-generating
analyzer that will be used
to construct the Query object used in highlighting. The resulting object,
in my case, is something like this:

  TermQuery: "audit trail"

Note that my kwic field was not analyzed with word bigrams, though,
so although it contains "audit" and "trail" as adjacent tokens, it
does not contain the composite token "audit trail". As such, when
this TermQuery is used for highlighting, no snippets will be
generated. (This no-snippets situation probably depends on a few
other details of my situation that I haven't mentioned. But I'm trying
to avoid drowning people in details here.)

In contrast, let's suppose that, when doing my analysis for query
highlighting, I *ignore* which particular fields are being *searched*,
and instead use the query analyzer for the hl.fl field being requested. In
this case my hl.fl=kwic, and so the non-bigram analyzer will be used,
and so my Query object for highlight will be something like this:

  PhraseQuery: audit, trail

Unlike the earlier TermQuery, this PhraseQuery works fine for
highlighting with the kwic field, and generates nice snippets.

Does this example make any sense? It would probably be more helpful
to provide a test case, but I'll have to figure out how to make
one that would provide a compelling use case here but that will also run
without require you to download and apply patches from JIRA.

I've thrown together a patch that makes this highlighting analysis change, and
it doesn't seem to break the test suite in any major ways. I may put it up on
JIRA, but it's kind of a hack. What's more, *how* to make the change in behavior
I'm talking about is sort of a separate question from whether it's
wildly off course
even in theory.

What do you think?

Chris