You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "OlivierJaquemet (via GitHub)" <gi...@apache.org> on 2026/07/31 09:25:11 UTC
[I] MultiFieldQueryParser + per-field boosts produce wrong Occur under AND default operator [lucene]
OlivierJaquemet opened a new issue, #16441:
URL: https://github.com/apache/lucene/issues/16441
### Description
## Disclaimer
I found this bug in our codebase and manually wrote the unit tests covering the various cases. I then used Claude AI to help identify the root cause and draft this report. Finally, I verified both the reported behavior and the proposed patch against Lucene's main branch (running all tests) before submitting this report and corresponding merge request.
I could not find any policy in the [Contributing to Lucene Guide](https://github.com/apache/lucene/blob/main/CONTRIBUTING.md) preventing me from going this way.
My apologies if this is not authorized.
## Summary
When `MultiFieldQueryParser` is configured with per-field boosts and the default operator is set to `AND`, a single-term query whose text happens to be sent through the "multi-term" (whitespace-splitting) code path produces a query where **every field clause is forced to `MUST`**, even though the parser is expected to keep the per-field alternatives as `SHOULD` inside an optional group. Without boosts, the same input produces the expected (unforced) result.
**Correct behavior with "simple" term query (no boost):**
```
MultiFieldQueryParser parser = new MultiFieldQueryParser(new String[] {"field1", "field2"}, new StandardAnalyzer());
parser.setDefaultOperator(QueryParser.Operator.AND);
assertEquals("field1:hello field2:hello",
parser.parse(QueryParser.escape("hello !")).toString());
```
**Incorrect behavior with boosted query:**
```
MultiFieldQueryParser parser2 = new MultiFieldQueryParser(new String[] {"field1", "field2"}, new StandardAnalyzer(), Map.of("field1", 2.0f));
parser2.setDefaultOperator(QueryParser.Operator.AND);
assertEquals("field1:hello field2:hello",
parser2.parse(QueryParser.escape("hello !")).toString()); // KO : +(field1:hello)^2.0 +field2:hello
```
# Root cause
`QueryParser.escape("hello !")` escapes the `!`, so the query grammar sees two
adjacent `TERM` tokens (`hello`, `\!`) rather than a `NOT` operator. With
`splitOnWhitespace=false` (the default), this hits the `MultiTerm` production in
`QueryParser.jj`, which concatenates the tokens into a single string and calls:
```java
firstQuery = getFieldQuery(field, discardEscapeChar(text.image), false);
addMultiTermClauses(clauses, firstQuery);
```
`StandardAnalyzer` reduces `"hello !"` to the single token `hello` (the `!` is
punctuation and is dropped), so `MultiFieldQueryParser.getFieldQuery(null, "hello !", false)`
returns a flat `BooleanQuery` combining one `SHOULD` clause per field:
- **Without boost:** `BooleanQuery(SHOULD(TermQuery field1:hello), SHOULD(TermQuery field2:hello))`
- **With boost:** `BooleanQuery(SHOULD(BoostQuery(TermQuery field1:hello, 2.0)), SHOULD(TermQuery field2:hello))`
The bug is in `QueryParserBase#addMultiTermClauses`:
```java
protected void addMultiTermClauses(List<BooleanClause> clauses, Query q) {
if (q == null) return;
boolean allNestedTermQueries = false;
if (q instanceof BooleanQuery) {
allNestedTermQueries = true;
for (BooleanClause clause : ((BooleanQuery) q).clauses()) {
if (!(clause.getQuery() instanceof TermQuery)) {
allNestedTermQueries = false;
break;
}
}
}
if (allNestedTermQueries) {
clauses.addAll(((BooleanQuery) q).clauses()); // keeps original Occur
} else {
BooleanClause.Occur occur = operator == OR_OPERATOR ? SHOULD : MUST;
if (q instanceof BooleanQuery) {
for (BooleanClause clause : ((BooleanQuery) q).clauses()) {
clauses.add(newBooleanClause(clause.getQuery(), occur)); // overwrites Occur
}
} else {
clauses.add(newBooleanClause(q, occur));
}
}
}
```
The `allNestedTermQueries` heuristic is meant to detect "this `BooleanQuery` is just
the result of the analyzer splitting one field's text into several terms", in which
case the original per-clause `Occur` is preserved as-is. It uses a strict
`clause.getQuery() instanceof TermQuery` check.
- Without a boost, every clause is a bare `TermQuery` → the check passes → the
original `SHOULD` occurs are copied through untouched (this happens to look correct
here, though it also means the default operator is silently ignored for this path
in general — a separate, pre-existing quirk).
- With a boost, the `field1` clause becomes a `BoostQuery` wrapping a `TermQuery`, so
`instanceof TermQuery` is `false` → the check fails → the code falls into the `else`
branch. Because `q` is still a `BooleanQuery`, it is unwrapped and **every nested
clause is re-added with a single, forced `Occur`** (`MUST` here, since the default
operator is `AND`), discarding the `SHOULD` semantics that expressed "any of these
fields matching is enough". The result: two independently required clauses instead
of an optional group with boosts.
In short: `addMultiTermClauses` cannot distinguish "a nested `BooleanQuery` that
happens to contain something more than a bare `TermQuery` (e.g. a boosted term)" from
"a nested `BooleanQuery` that legitimately needs its per-clause structure discarded
and replaced by a single shared `Occur`" (e.g. synonym expansion). The multi-field
disjunction built by `MultiFieldQueryParser` falls victim to the latter path purely
because one of its clauses is boosted.
## Impact
Any use of `MultiFieldQueryParser` with per-field boosts, `AND` as the default operator, and query text that is analyzed down to a single term while going through the whitespace-splitting (`MultiTerm`) grammar path will silently produce an over-restrictive query (`MUST` on every field) instead of the intended optional/boosted disjunction. This can happen with escaped punctuation (as above), but also with any input where the analyzer collapses multiple raw tokens into one term (stop-word removal, synonym filtering that drops a token, etc.), so it is not limited to this specific escaping scenario.
### Version and environment details
Affected version: main and 10.4.0
Component: lucene-queryparser (classic package)
Affected classes:
- org.apache.lucene.queryparser.classic.MultiFieldQueryParser,
- org.apache.lucene.queryparser.classic.QueryParserBase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] MultiFieldQueryParser + per-field boosts produce wrong Occur under AND default operator [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna closed issue #16441: MultiFieldQueryParser + per-field boosts produce wrong Occur under AND default operator
URL: https://github.com/apache/lucene/issues/16441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] MultiFieldQueryParser + per-field boosts produce wrong Occur under AND default operator [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on issue #16441:
URL: https://github.com/apache/lucene/issues/16441#issuecomment-6418171747
Fixed by #16446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org