Missing highlights with AND-queries and unified highlighter (Solr 8.11.1 and 9.2.0)

[Morten Ernebjerg <mo...@data4life.care.INVALID>]

Hi there

I have a curious problem with the unified highlighter (both for Solr 8.11.1
and 9.2.0). Specifically, if I

1. do a Boolean AND-query on an indexed field filled by copyFields with
input from two underlying fields in the raw document, and
2. the two ANDed terms match entries from different source fields in the
same document, and
3. I request highlighting on the two source fields

then the unified matcher does not return highlights for the matching terms.
Interestingly, the original highlighter *does* return the expected
highlights, but going back to the original highlighter is not a real option
for other reasons, so I'm hoping someone will know a solution.

The simplest set-up to recreate the problem on using the _default configset
(tested on 9.2.0 but probably the same on 8.11.1) is this:

1. Define the following fields in the schema:
- "title": (field type=text_general, stored, tokenized, multivalued, not
indexed)
- "description": (same settings as "title")
- "search": (text type=text_general,  indexed, multivalued, uninvertible,
not stored)
- Copy fields from "title" and "description" to "search"

2. Upload the following document
{
    "id": "1",
    "title": "A fantastic title",
    "description": "A wonderful description"
}

3. Do the search "search:fantastic AND search:wonderful" - as expected, the
document is returned (I'm using the edismax parser, but seems to be the
same for the Lucene one).

4. Switch on highlighting for the fields "title" and "description" (using
hl.fl). I expect one highlight in each field, one for each search term.
However, with the unified highlighter (default in 9.2), I get zero
highlights - more specifically, I get an empty entry in the "highlighting"
property in the JSON response:

"highlighting": {
    "1": {}
}

If I choose the original highlighter (hl.method=original), I get the
expected two highlights. If I do a similar search where both matches are in
the *same* field (e.g. "search:fantastic AND search:title"), I get
highlights with both highlighters. The same is true if I do the original
query but with OR instead of AND ("search:fantastic OR search:wonderful").

I played around a bit with hl.requireFieldMatch and hl.queryFieldPattern
but did not find a combination that fixed the issue.

Cheers & thanks in advance,

Morten


--

*Morten Ernebjerg, Ph.D.*

Senior Developer


morten.ernebjerg@data4life.care

D4L data4life gGmbH
c/o Digital Health Cluster (DHC) im Hasso-Plattner-Institut (HPI)
Rudolf-Breitscheid-Straße 187
14482 Potsdam, Germany

www.data4life.care

Amtsgericht Potsdam, HRB 30667

Managing Director: Christian-Cornelius Weiß


We are Data4Life. We've been certified by the German Federal Office for
Information Security (BSI) in accordance with ISO 27001 on the basis of
"IT-Grundschutz".





<https://www.data4life.care/en/studies/daki-health/?mtm_campaign=221020_daki-emailbanner&mtm_kwd=landingpage&mtm_source=signaturebanner&mtm_medium=email&mtm_content=en&mtm_cid=dakihealth>