You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by jo...@apache.org on 2026/09/29 11:00:24 UTC
(jackrabbit-oak) branch oak-search-mongot updated: OAK-search-mongot: cut per-query round trips for the next POC build (#3170)
This is an automated email from the ASF dual-hosted git repository.
Joscorbe pushed a commit to branch oak-search-mongot
in repository https://gitbox.apache.org/repos/asf/jackrabbit-oak.git
The following commit(s) were added to refs/heads/oak-search-mongot by this push:
new 565ecf15ea OAK-search-mongot: cut per-query round trips for the next POC build (#3170)
565ecf15ea is described below
commit 565ecf15eac81cf1d5696196d7f5051244d63850
Author: Ming He <me...@gmail.com>
AuthorDate: Tue Sep 29 04:00:17 2026 -0700
OAK-search-mongot: cut per-query round trips for the next POC build (#3170)
* OAK-search-mongot: let the connector bundle resolve in OSGi
OSGiIT.bundleStates has failed since the connector was added to oak-it-osgi:
the bundle stays INSTALLED because Felix cannot wire its com.mongodb.* and
org.bson.* imports.
The connector must inline oak-search, which exports no packages, as oak-lucene
and oak-search-elastic do. The additional `*;scope=compile|runtime` embed only
pulled in mongodb-driver-sync, not the driver-core and bson bundles it depends
on, so the driver imports remained unresolved; it also embedded oak-search a
second time as a nested jar.
Stop embedding the driver and deploy the MongoDB driver bundles in the OSGi
test runtime instead, the way oak-store-document consumes the driver. The
connector bundle now has no embedded jars and imports com.mongodb and org.bson
in [5.4,6). The README still described importing oak-search packages and
driver 5.3; correct it.
* OAK-search-mongot: drop per-query search index lookups and redundant sorts
Three pieces of per-query work never change the results:
- Every $search aggregation first listed the collection's search indexes,
one extra round trip per query and two for an exact size. $search against
a missing index returns no hits instead of failing, so keep the check for
that case only: run it when a $search has no hits, and while retrying an
index that is not yet queryable so a missing or failed index still fails
fast.
- The exact-size count appended $count to the full result pipeline,
including the score $set, a blocking $sort and the $project. Drop those
stages before counting.
- order by [jcr:score] desc added a $set and a blocking $sort of every hit,
although $search already returns hits by descending score and the
following $match stages keep that order.
The new core compatibility checks count the stages the server runs, since
results are identical either way; they fail against the previous code. A
new reindex compatibility test makes sure a dropped search index still fails
the query instead of returning no results.
* OAK: count matches via search metadata instead of a re-executed $count pipeline
A $count stage appended after $search (with $match/$sort in between) makes
mongod re-execute the query and drain every matching document from mongot just
to count it - the documented $count-after-$search anti-pattern. When the
pipeline is a bare [$search, $project] pair, ask mongot for the count through
the count: {type: "total"} search option instead: it is evaluated index-side
and returned as a single search-metadata document. Keep the $count pipeline as
a fallback when post-search stages ($match filters, $sort) would make the
metadata count inexact.
* OAK-search-mongot: count matches inside mongot through the search metadata
Follow-up to #3143, complements #3159. Covers the exact-size case #3159
explicitly left for a follow-up PR.
* OAK-search-mongot: cache the index statistics Oak reads while planning
Oak's planner reads numDocs() and getDocCountFor(property) for every query.
Each call ran countDocuments against the index collection, so every query
paid two or three extra round trips to MongoDB before it started, and the
property counts scanned the collection because nothing indexes those fields.
Serve the counts from a cache, as the Elasticsearch index does: entries are
refreshed in the background after 60 seconds and expire after 10 minutes
(oak.mongot.statsRefreshSeconds, oak.mongot.statsExpireSeconds). The cache
belongs to the index node, which the tracker replaces whenever the index
status or definition changes, so each indexing cycle that writes to the
index also starts from fresh counts.
* OAK-search-mongot: observe test queries through a client command listener
The query regression tests counted aggregation stages through
serverStatus.metrics.aggStageCounters. That needs the clusterMonitor role
and counts every client on the server, so parallel work skews the numbers.
Record the aggregate commands this connector sends with a driver
CommandListener instead. MongoConnection gains a package-private factory
that takes client settings, so tests can attach the listener.
* OAK-search-mongot: document the source-built connector and the current suite
The previous commit also removed the prebuilt 2.5-SNAPSHOT jar from dist/:
it predates the current packaging and would install a connector that no
longer matches this source. The README now points at the jar the module
build produces.
The qualification numbers came from the original POC, which ran against a
patched Mongot, and predate the class-level @Ignore on
MongotFullTextAnalyzerCommonTest. Describe the suite as it runs against the
stock atlas-local server: the analyzer class is skipped because 15 of its
55 contracts use analysis components Mongot rejects, and
PropertyIndexCommonTest runs 21 tests.
---------
Co-authored-by: oren.ovadia <or...@mongodb.com>
---
oak-it-osgi/test-bundles.xml | 3 +
oak-search-mongot/COMPATIBILITY.md | 17 +++--
oak-search-mongot/README.md | 35 ++++------
oak-search-mongot/dist/README.md | 43 ------------
.../dist/oak-search-mongot-2.5-SNAPSHOT.jar | Bin 121460 -> 0 bytes
oak-search-mongot/pom.xml | 6 +-
.../oak/plugins/index/mongot/MongoConnection.java | 6 ++
.../index/mongot/MongotIndexStatistics.java | 33 +++++++--
.../plugins/index/mongot/query/MongotIndex.java | 76 ++++++++++++++++++---
.../index/mongot/MongotCommandRecorder.java | 56 +++++++++++++++
.../index/mongot/MongotIndexStatisticsTest.java | 56 +++++++++++++++
.../index/mongot/MongotTestRepositoryBuilder.java | 13 +++-
.../query/MongotCoreQueryCompatibilityTest.java | 45 ++++++++++++
.../query/MongotReindexCompatibilityTest.java | 21 ++++++
14 files changed, 313 insertions(+), 97 deletions(-)
diff --git a/oak-it-osgi/test-bundles.xml b/oak-it-osgi/test-bundles.xml
index 6682398b2a..29d17c1ceb 100644
--- a/oak-it-osgi/test-bundles.xml
+++ b/oak-it-osgi/test-bundles.xml
@@ -58,6 +58,9 @@
<include>org.apache.jackrabbit:oak-lucene</include>
<include>org.apache.jackrabbit:oak-search-elastic</include>
<include>org.apache.jackrabbit:oak-search-mongot</include>
+ <include>org.mongodb:bson</include>
+ <include>org.mongodb:mongodb-driver-core</include>
+ <include>org.mongodb:mongodb-driver-sync</include>
<include>org.apache.tika:tika-core</include>
<include>org.apache.jackrabbit:oak-blob</include>
<include>org.apache.jackrabbit:oak-blob-cloud-azure</include>
diff --git a/oak-search-mongot/COMPATIBILITY.md b/oak-search-mongot/COMPATIBILITY.md
index 3fa9326c17..566c455b4b 100644
--- a/oak-search-mongot/COMPATIBILITY.md
+++ b/oak-search-mongot/COMPATIBILITY.md
@@ -31,16 +31,15 @@ The clean release gate is:
mvn -pl oak-search-mongot clean verify
```
-The qualified build reports **468 tests, 0 failures, 0 errors, and 4 inherited
-skips**. There are no connector-owned `@Ignore` methods and no opt-in failing
+Against the stock `mongodb/mongodb-atlas-local` test server, the build reports
+**0 failures, 0 errors, and 4 skips**. There is no opt-in failing
qualification lane.
-The four skips come from the upstream Oak test sources rather than this
-connector:
+The skips are:
-| Upstream test | Why it cannot qualify this backend |
+| Skipped test | Why it cannot qualify this backend |
| --- | --- |
-| `analyzerWithHyphenationCompoundWord` | The inherited fixture contains literal placeholder XML (`<...>`) and is already ignored upstream. |
+| `MongotFullTextAnalyzerCommonTest` (whole class) | Ignored at class level: 15 of its 55 inherited analyzer contracts configure Lucene token or character filters, stemmers, or options that stock Mongot rejects when the search index is created. Surefire reports the class as one skip. |
| `ambiguousSubtreeIndexWithDescendantConstraint` | OAK-3992 hard-codes Lucene-specific plan behavior before backend execution. |
| `unionOnTwoDescendants` | OAK-3993 is rejected by Oak's shared SQL2 parser before an index plugin is invoked. |
| `descendantTestWithIndexTagExplainWithNoData` | The shared Oak planner chooses traversal for an empty index despite the tag; the inherited test is already ignored with that explanation. |
@@ -127,7 +126,7 @@ tests are supporting diagnostics, not the compatibility verdict.
| Facets | SUPPORTED | All 14 inherited secure, insecure, and statistical cases execute and pass. |
| Excerpts and highlighting | SUPPORTED | All 9 excerpt cases plus the inherited aggregate excerpt contract pass. |
| Suggestions and spellcheck | SUPPORTED | Base and descendant contracts pass; only the two upstream OAK-3992/OAK-3993 tests remain skipped. |
-| Built-in and composed analyzers | SUPPORTED | 54 executable inherited analyzer contracts pass; the one upstream placeholder fixture remains skipped. |
+| Built-in and composed analyzers | PARTIAL | The inherited analyzer suite is ignored as a class: 15 of its 55 contracts use analysis components that stock Mongot rejects. |
| Dynamic boost | SUPPORTED | All 12 inherited cases pass, including analyzed terms, exclusions, confidence ordering, and similarity-tag reranking. |
| Similarity and vectors | SUPPORTED FOR POC | SQL2/XPath similarity, more-like-this, vector mapping/storage/ranking, lexical filtering, and explain output pass. Candidate and relevance tuning remain workload work, not missing plumbing. |
| Updates, deletes, moves, reindex | SUPPORTED | Incremental mutation, subtree deletion, move, aggregate refresh, long-path identity, stale-document exclusion, and atomic generation handoff tests pass. Superseded-generation deletion is deliberately deferred to a future cleaner. |
@@ -143,7 +142,7 @@ tests are supporting diagnostics, not the compatibility verdict.
| Common suite | Result |
| --- | --- |
| `FullTextIndexCommonTest` | 14/14 pass |
-| `PropertyIndexCommonTest` | 20/20 pass |
+| `PropertyIndexCommonTest` | 21/21 pass |
| `IndexPathRestrictionCommonTest` | 7/7 pass |
| `OrderByCommonTest` | 14/14 pass |
| `FunctionIndexCommonTest` | 21/21 pass |
@@ -155,7 +154,7 @@ tests are supporting diagnostics, not the compatibility verdict.
| `ExcerptTest` | 9/9 pass |
| Suggestion suites | 19 pass, 2 upstream skips |
| Spellcheck suites | 10/10 pass |
-| `FullTextAnalyzerCommonTest` | 54 pass, 1 upstream skip |
+| `FullTextAnalyzerCommonTest` | Not run: ignored as a class (1 skip). With the ignore removed, 39 pass, 15 error because stock Mongot rejects their analyzers, and 1 upstream skip |
| `DynamicBoostCommonTest` | 12/12 pass |
| Result-size suites | 2/2 pass |
| `IndexImporterReindexTest` | 1/1 pass |
diff --git a/oak-search-mongot/README.md b/oak-search-mongot/README.md
index 6aaa9a76db..c3676908a0 100644
--- a/oak-search-mongot/README.md
+++ b/oak-search-mongot/README.md
@@ -111,9 +111,11 @@ Run the clean qualification gate with the Atlas SRV connection string:
Use the Atlas-generated SRV connection string without
`directConnection=true`.
-The clean gate reports 468 tests, 0 failures, 0 errors, and 4 inherited Oak
-skips, including the importer, commit, generation, and streaming contracts.
-There are no connector-owned ignored tests or expected-failure lane.
+The clean gate reports 0 failures, 0 errors, and 4 skips, including the
+importer, commit, generation, and streaming contracts. The skips are the
+inherited analyzer suite, ignored as a class because stock Mongot rejects some
+of its analysis components, and three inherited Oak skips; see
+[COMPATIBILITY.md](COMPATIBILITY.md). There is no expected-failure lane.
Focused end-to-end query evidence:
@@ -147,21 +149,15 @@ The module is packaged as an OSGi bundle:
mvn -pl oak-search-mongot -am -DskipTests package
-The deployable artifact is:
-
- oak-search-mongot/target/oak-search-mongot-2.5-SNAPSHOT.jar
-
-A prebuilt copy from this source tree is checked in at
-[dist/oak-search-mongot-2.5-SNAPSHOT.jar](dist/oak-search-mongot-2.5-SNAPSHOT.jar).
-Its bundle identity, checksum, and runtime requirements are recorded in the
-[distribution notes](dist/README.md).
+The deployable artifact is `oak-search-mongot/target/oak-search-mongot-<version>.jar`,
+where `<version>` is the Oak version of this source tree.
Install that bundle into a compatible Oak OSGi runtime, together with the
-runtime bundles it imports. The current build expects the matching Oak API and
-`oak-search` packages plus the MongoDB Java Driver 5.3 bundle set
-(`mongodb-driver-sync`, `mongodb-driver-core`, `bson`, and
-`bson-record-codec`); those dependencies are not embedded in the connector
-JAR. Do not install the `*-tests.jar` artifact.
+runtime bundles it imports. The bundle embeds the `oak-search` classes it uses,
+because `oak-search` exports no packages, and imports the matching Oak API
+packages plus the MongoDB Java Driver 5.x bundle set (`mongodb-driver-sync`,
+`mongodb-driver-core`, and `bson`, version 5.4 or later); the driver is not
+embedded in the connector JAR. Do not install the `*-tests.jar` artifact.
Configure the component PID after installing the bundle:
@@ -190,11 +186,8 @@ property `type=mongot`.
Evaluator delivery
------------------
-This repository includes both the [connector source](src/main/java/) and a
-prebuilt, version-matched
-[OSGi bundle](dist/oak-search-mongot-2.5-SNAPSHOT.jar), so the initial
-evaluation does not require packaging the connector. The source is available
-for inspection and for rebuilding against a different Oak baseline.
+Build the connector bundle from this source tree, as described above, so that
+it matches the Oak version it is deployed with.
MongoDB provisions the compatible custom Mongot build behind the Atlas cluster;
no Mongot binary or configuration is distributed to the evaluator. The source
qualification suite requires the repository and Atlas access. Runtime use
diff --git a/oak-search-mongot/dist/README.md b/oak-search-mongot/dist/README.md
deleted file mode 100644
index befba2b90a..0000000000
--- a/oak-search-mongot/dist/README.md
+++ /dev/null
@@ -1,43 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-Connector bundle
-================
-
-`oak-search-mongot-2.5-SNAPSHOT.jar` is the prebuilt OSGi bundle for this
-[source tree](..). Install this file, not the `*-tests.jar` produced by Maven.
-
-Bundle identity:
-
-- symbolic name: `org.apache.jackrabbit.oak-search-mongot`
-- version: `2.5.0.SNAPSHOT`
-- Java execution environment: Java 17 or newer
-- SHA-256: `2996ff5aee0108c8b6e5a81c9dc289ef4eb2317663700266040a6e7adc48d55f`
-
-The bundle intentionally does not embed Oak or MongoDB driver classes. The
-target OSGi runtime must provide the compatible Oak 2.5 packages and MongoDB
-Java Driver 5.3 bundle set: `mongodb-driver-sync`, `mongodb-driver-core`,
-`bson`, and `bson-record-codec`.
-
-To reproduce the bundle from the repository root:
-
- mvn -pl oak-search-mongot -am -DskipTests package
-
-The resulting file is
-`oak-search-mongot/target/oak-search-mongot-2.5-SNAPSHOT.jar`. See the module
-[README](../README.md#build-and-install-the-connector-bundle) for installation
-and OSGi configuration instructions.
diff --git a/oak-search-mongot/dist/oak-search-mongot-2.5-SNAPSHOT.jar b/oak-search-mongot/dist/oak-search-mongot-2.5-SNAPSHOT.jar
deleted file mode 100644
index d0fc019d7b..0000000000
Binary files a/oak-search-mongot/dist/oak-search-mongot-2.5-SNAPSHOT.jar and /dev/null differ
diff --git a/oak-search-mongot/pom.xml b/oak-search-mongot/pom.xml
index 21aece553d..6faf930c07 100644
--- a/oak-search-mongot/pom.xml
+++ b/oak-search-mongot/pom.xml
@@ -43,11 +43,7 @@
<configuration>
<instructions>
<Export-Package>!*</Export-Package>
- <Embed-Dependency>
- oak-search;scope=compile|runtime;inline=true,
-
- *;scope=compile|runtime
- </Embed-Dependency>
+ <Embed-Dependency>oak-search;scope=compile|runtime;inline=true</Embed-Dependency>
</instructions>
</configuration>
</plugin>
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongoConnection.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongoConnection.java
index 693684a730..0e46e0942d 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongoConnection.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongoConnection.java
@@ -16,6 +16,7 @@
*/
package org.apache.jackrabbit.oak.plugins.index.mongot;
+import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
@@ -36,6 +37,11 @@ public final class MongoConnection implements AutoCloseable {
return new MongoConnection(MongoClients.create(connectionString), databaseName);
}
+ // Lets tests attach a command listener and observe only the connector's own commands.
+ static MongoConnection create(MongoClientSettings settings, String databaseName) {
+ return new MongoConnection(MongoClients.create(settings), databaseName);
+ }
+
public MongoDatabase getDatabase() {
return database;
}
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatistics.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatistics.java
index 2f92a125e5..b3d4ab108a 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatistics.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatistics.java
@@ -16,34 +16,55 @@
*/
package org.apache.jackrabbit.oak.plugins.index.mongot;
+import java.time.Duration;
+
+import org.apache.jackrabbit.oak.cache.api.CacheBuilder;
+import org.apache.jackrabbit.oak.cache.api.LoadingCache;
import org.apache.jackrabbit.oak.plugins.index.mongot.index.MongoFieldNames;
import org.apache.jackrabbit.oak.plugins.index.search.IndexStatistics;
import org.apache.jackrabbit.oak.plugins.index.search.FieldNames;
+import org.bson.Document;
import static com.mongodb.client.model.Filters.exists;
public final class MongotIndexStatistics implements IndexStatistics {
+ private static final long MAX_SIZE = Long.getLong("oak.mongot.statsMaxSize", 10_000L);
+ private static final long EXPIRE_SECONDS = Long.getLong("oak.mongot.statsExpireSeconds", 10 * 60L);
+ private static final long REFRESH_SECONDS = Long.getLong("oak.mongot.statsRefreshSeconds", 60L);
+ private static final String ALL_DOCUMENTS = "";
+
private final MongoConnection connection;
private final MongotIndexDefinition definition;
+ // Oak's planner reads these counts for every query. Like the Elasticsearch index, serve
+ // them from a cache so planning does not cost a round trip to MongoDB per count.
+ private final LoadingCache<String, Integer> counts;
MongotIndexStatistics(MongoConnection connection, MongotIndexDefinition definition) {
this.connection = connection;
this.definition = definition;
+ this.counts = CacheBuilder.<String, Integer>newBuilder()
+ .maximumSize(MAX_SIZE)
+ .expireAfterWrite(Duration.ofSeconds(EXPIRE_SECONDS))
+ .refreshAfterWrite(Duration.ofSeconds(REFRESH_SECONDS))
+ .build(this::countDocuments);
}
@Override
public int numDocs() {
- return (int) Math.min(Integer.MAX_VALUE,
- connection.getCollection(definition).countDocuments());
+ return counts.get(ALL_DOCUMENTS);
}
@Override
public int getDocCountFor(String key) {
- String field = FieldNames.NULL_PROPS.equals(key)
+ return counts.get(FieldNames.NULL_PROPS.equals(key)
? MongoFieldNames.NULL_PROPERTIES
- : MongoFieldNames.TYPED + "." + MongoFieldNames.encodeProperty(key);
- return (int) Math.min(Integer.MAX_VALUE,
- connection.getCollection(definition).countDocuments(exists(field)));
+ : MongoFieldNames.TYPED + "." + MongoFieldNames.encodeProperty(key));
+ }
+
+ private int countDocuments(String field) {
+ long count = connection.getCollection(definition).countDocuments(
+ ALL_DOCUMENTS.equals(field) ? new Document() : exists(field));
+ return (int) Math.min(Integer.MAX_VALUE, count);
}
}
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
index 8ad18dba1d..1fc996b7fa 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
@@ -25,6 +25,7 @@ import java.util.List;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.function.Predicate;
+import java.util.stream.Collectors;
import com.mongodb.MongoCommandException;
import com.mongodb.client.AggregateIterable;
@@ -223,13 +224,13 @@ final class MongotIndex extends FulltextIndex {
MongotResultAdapter.excerpts(result, request.excerptColumns()),
null, null);
}, documents::close);
- // Use a query-specific $count aggregation instead of rows::getSize: the latter
- // would fully drain the streaming cursor (in aggregateCursor()'s batches) the
- // moment anything calls Cursor.getSize()/JCR's RowIterator.getSize() - a common
+ // Use a query-specific count instead of rows::getSize: the latter would fully
+ // drain the streaming cursor (in aggregateCursor()'s batches) the moment
+ // anything calls Cursor.getSize()/JCR's RowIterator.getSize() - a common
// "N results found" UI pattern - even when the caller never intends to iterate
- // all rows. $count is cheap (server-side count, no document bodies returned)
- // and, unlike the unused getSizeEstimator(IndexPlan) override above, reflects
- // this query's actual filter rather than the whole index's document count.
+ // all rows. countMatches() asks mongot for the match count through the search
+ // metadata rather than appending a $count stage, which would re-execute the
+ // query and drain every matching document from mongot into mongod.
SizeEstimator sizeEstimator = () -> countMatches(collection, pipeline, definition);
return new FulltextPathCursor(rows, NEVER_REWOUND, plan,
plan.getFilter().getQueryLimits(), sizeEstimator);
@@ -238,13 +239,56 @@ final class MongotIndex extends FulltextIndex {
private static long countMatches(MongoCollection<Document> collection,
List<Document> pipeline,
MongotIndexDefinition definition) {
- List<Document> countPipeline = new ArrayList<>(pipeline);
+ Document search = searchOnlyPipeline(pipeline);
+ if (search != null) {
+ // The count option is evaluated inside mongot and returned through the
+ // search metadata as a single document, without fetching or sorting any
+ // matching documents. Appending a $count stage to the pipeline instead
+ // would re-execute the query and stream every match from mongot into
+ // mongod just to count it.
+ Document searchMeta = new Document("$searchMeta",
+ new Document(search).append("count",
+ new Document("type", "total")));
+ try (MongoCursor<Document> cursor = aggregateCursor(collection,
+ List.of(searchMeta), definition)) {
+ if (cursor.hasNext()) {
+ Document count = cursor.next().get("count", Document.class);
+ if (count != null && count.get("total") instanceof Number total) {
+ return total.longValue();
+ }
+ }
+ return 0L;
+ }
+ }
+ // The plan has stages after $search ($match filters or $sort) whose effect the
+ // search metadata cannot see, so count with a trailing $count stage. This
+ // still runs in mongod against the full result stream; pushing those stages
+ // into the search operator would let this path use the metadata count too.
+ // Sorting and projecting do not change the count, and $sort blocks on every hit.
+ List<Document> countPipeline = pipeline.stream()
+ .filter(stage -> !stage.containsKey("$sort") && !stage.containsKey("$set")
+ && !stage.containsKey("$project"))
+ .collect(Collectors.toCollection(ArrayList::new));
countPipeline.add(new Document("$count", "n"));
try (MongoCursor<Document> cursor = aggregateCursor(collection, countPipeline, definition)) {
return cursor.hasNext() ? ((Number) cursor.next().get("n")).longValue() : 0L;
}
}
+ /**
+ * Returns the {@code $search} stage when the pipeline is a bare
+ * {@code [$search, $project]} pair, i.e. when no post-search stage filters or
+ * reorders the matches and a metadata count would be exact; {@code null} otherwise.
+ */
+ private static Document searchOnlyPipeline(List<Document> pipeline) {
+ if (pipeline.size() == 2
+ && pipeline.get(0).containsKey("$search")
+ && pipeline.get(1).containsKey("$project")) {
+ return (Document) pipeline.get(0).get("$search");
+ }
+ return null;
+ }
+
private static Cursor cursor(List<FulltextResultRow> rows, IndexPlan plan) {
Iterator<FulltextResultRow> iterator = rows.iterator();
return new FulltextPathCursor(iterator, NEVER_REWOUND, plan,
@@ -284,9 +328,6 @@ final class MongotIndex extends FulltextIndex {
List<Document> pipeline,
MongotIndexDefinition definition) {
boolean usesSearch = !pipeline.isEmpty() && pipeline.get(0).containsKey("$search");
- if (usesSearch) {
- requireSearchIndex(collection, definition.getSearchIndexName());
- }
long deadline = System.nanoTime()
+ TimeUnit.MILLISECONDS.toNanos(SEARCH_READINESS_TIMEOUT_MILLIS);
while (true) {
@@ -316,11 +357,19 @@ final class MongotIndex extends FulltextIndex {
AggregateIterable<Document> aggregation = collection.aggregate(pipeline)
.batchSize(batchSize)
.maxTime(definition.getQueryTimeoutMillis(), TimeUnit.MILLISECONDS);
- return aggregation.iterator();
+ MongoCursor<Document> cursor = aggregation.iterator();
+ // $search against a missing index returns no hits instead of failing, so
+ // confirm the index exists only when there are none, not on every query.
+ if (usesSearch && !cursor.hasNext()) {
+ requireSearchIndex(collection, definition.getSearchIndexName());
+ }
+ return cursor;
} catch (MongoCommandException e) {
if (!isSearchIndexStarting(e) || System.nanoTime() >= deadline) {
throw e;
}
+ // Fail fast when the index is missing or failed instead of waiting for it.
+ requireSearchIndex(collection, definition.getSearchIndexName());
try {
TimeUnit.MILLISECONDS.sleep(SEARCH_READINESS_RETRY_MILLIS);
} catch (InterruptedException interrupted) {
@@ -436,6 +485,11 @@ final class MongotIndex extends FulltextIndex {
requiredSortValues.append(field, new Document("$exists", true));
}
}
+ if (sort.equals(new Document("_score", -1))) {
+ // $search already returns hits by descending score; sorting again would block.
+ sort.clear();
+ scoreMaterialized = false;
+ }
if (!sort.isEmpty()) {
// Oak's ordered indexes only expose rows that have a value for every
// sort key. MongoDB sorts documents with missing fields as null, so
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotCommandRecorder.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotCommandRecorder.java
new file mode 100644
index 0000000000..e214530954
--- /dev/null
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotCommandRecorder.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.index.mongot;
+
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+
+import com.mongodb.ConnectionString;
+import com.mongodb.MongoClientSettings;
+import com.mongodb.event.CommandListener;
+import com.mongodb.event.CommandStartedEvent;
+import org.bson.BsonDocument;
+
+/**
+ * Records the aggregations a connector sends through its own client, so tests observe
+ * only the connector's commands and need no server-wide privileges.
+ */
+final class MongotCommandRecorder implements CommandListener {
+
+ private final List<BsonDocument> aggregations = new CopyOnWriteArrayList<>();
+
+ MongoConnection connect(String connectionString, String databaseName) {
+ return MongoConnection.create(MongoClientSettings.builder()
+ .applyConnectionString(new ConnectionString(connectionString))
+ .addCommandListener(this)
+ .build(), databaseName);
+ }
+
+ @Override
+ public void commandStarted(CommandStartedEvent event) {
+ if ("aggregate".equals(event.getCommandName())) {
+ aggregations.add(event.getCommand().clone());
+ }
+ }
+
+ long stageCount(String stage) {
+ return aggregations.stream()
+ .filter(command -> command.getArray("pipeline").stream()
+ .anyMatch(step -> step.asDocument().containsKey(stage)))
+ .count();
+ }
+}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatisticsTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatisticsTest.java
new file mode 100644
index 0000000000..7fbc0e53da
--- /dev/null
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexStatisticsTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.index.mongot;
+
+import java.util.List;
+
+import org.apache.jackrabbit.oak.plugins.index.mongot.index.MongoDocument;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+public class MongotIndexStatisticsTest {
+
+ @ClassRule
+ public static final MongotSearchConnectionRule mongo = new MongotSearchConnectionRule();
+
+ @Test
+ public void planningCountsQueryMongoDBOncePerCount() throws Exception {
+ MongotIndexDefinition definition = MongotIndexDefinitionTest.newDefinition("statistics");
+ MongoDocument published = new MongoDocument("/published");
+ published.addTypedProperty("status", "published");
+ mongo.getSearchConnection().getCollection(definition).insertMany(List.of(
+ published.toBson(), new MongoDocument("/draft").toBson()));
+ MongotCommandRecorder commands = new MongotCommandRecorder();
+
+ try (MongoConnection connection = commands.connect(
+ mongo.getConnectionString(), mongo.getDatabaseName())) {
+ MongotIndexStatistics statistics = new MongotIndexStatistics(connection, definition);
+
+ // countDocuments runs as an aggregation with a $group stage.
+ assertEquals(2, statistics.numDocs());
+ assertEquals(1, statistics.getDocCountFor("status"));
+ assertEquals(2, commands.stageCount("$group"));
+
+ // Oak's planner asks again for every query; those reads must not reach MongoDB.
+ assertEquals(2, statistics.numDocs());
+ assertEquals(1, statistics.getDocCountFor("status"));
+ assertEquals(2, commands.stageCount("$group"));
+ }
+ }
+}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotTestRepositoryBuilder.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotTestRepositoryBuilder.java
index d0fcb1bdb0..51720b1acd 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotTestRepositoryBuilder.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotTestRepositoryBuilder.java
@@ -83,7 +83,8 @@ public final class MongotTestRepositoryBuilder {
public Fixture build() throws Exception {
root.child("oak:index").setChildNode(INDEX_NAME, definition.build());
MemoryNodeStore store = new MemoryNodeStore(root.getNodeState());
- MongoConnection connection = MongoConnection.create(
+ MongotCommandRecorder commands = new MongotCommandRecorder();
+ MongoConnection connection = commands.connect(
mongo.getConnectionString(), mongo.getDatabaseName());
AsyncIndexUpdate async = new AsyncIndexUpdate("async", store,
new MongotIndexEditorProvider(connection, null));
@@ -102,7 +103,7 @@ public final class MongotTestRepositoryBuilder {
.with(new OpenSecurityProvider())
.with(new MongotIndexProvider(tracker))
.createContentRepository();
- return new Fixture(connection, repository.login(null, null), store, async, tracker);
+ return new Fixture(connection, commands, repository.login(null, null), store, async, tracker);
}
public static final class Fixture implements AutoCloseable {
@@ -112,8 +113,10 @@ public final class MongotTestRepositoryBuilder {
private final MemoryNodeStore store;
private final AsyncIndexUpdate async;
private final MongotIndexTracker tracker;
+ private final MongotCommandRecorder commands;
private Fixture(MongoConnection connection,
+ MongotCommandRecorder commands,
ContentSession session,
MemoryNodeStore store,
AsyncIndexUpdate async,
@@ -123,6 +126,7 @@ public final class MongotTestRepositoryBuilder {
this.store = store;
this.async = async;
this.tracker = tracker;
+ this.commands = commands;
}
public void mutate(Consumer<NodeBuilder> mutation) throws CommitFailedException {
@@ -137,6 +141,11 @@ public final class MongotTestRepositoryBuilder {
tracker.update(store.getRoot());
}
+ /** Number of aggregations this fixture's connector sent whose pipeline contains the stage. */
+ public long stageCount(String stage) {
+ return commands.stageCount(stage);
+ }
+
public MongoCollection<Document> collection() {
return connection.getCollection(indexDefinition());
}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
index b06b704f74..c9640799ed 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
@@ -118,6 +118,22 @@ public class MongotCoreQueryCompatibilityTest {
repository.paths(base + "isdescendantnode(s, '/content/site') order by s.[price]", "JCR-SQL2"));
}
+ @Test
+ public void exactSizeOfABareSearchCountsInsideMongotWithoutACountStage() throws Exception {
+ String query = "select [jcr:path] from [nt:unstructured] as s where "
+ + "contains(s.[jcr:title], 'mongodb')";
+ long searchMetaStages = stageCount("$searchMeta");
+ long countStages = stageCount("$count");
+
+ assertEquals(4, repository.query(query, "JCR-SQL2")
+ .getSize(Result.SizePrecision.EXACT, Long.MAX_VALUE));
+
+ // The count comes from the search metadata, computed inside mongot, rather
+ // than from a re-executed pipeline that streams every hit into mongod.
+ assertEquals(searchMetaStages + 1, stageCount("$searchMeta"));
+ assertEquals(countStages, stageCount("$count"));
+ }
+
@Test
public void orderingLimitOffsetAndResultSize() throws Exception {
String query = "select [jcr:path] from [nt:unstructured] as s where "
@@ -136,6 +152,24 @@ public class MongotCoreQueryCompatibilityTest {
assertTrue(plan, plan.contains("mongot:"));
}
+ @Test
+ public void exactSizeOfAnIndexSortedSearchNeedsNoIndexLookupOrSecondSort() throws Exception {
+ String query = "select [jcr:path] from [nt:unstructured] as s where "
+ + "contains(s.[jcr:title], 'mongodb') and isdescendantnode(s, '/content/site') "
+ + "order by [jcr:score]";
+ long indexLookups = stageCount("$listSearchIndexes");
+ long sorts = stageCount("$sort");
+ long counts = stageCount("$count");
+
+ assertEquals(3, repository.query(query, "JCR-SQL2")
+ .getSize(Result.SizePrecision.EXACT, Long.MAX_VALUE));
+
+ // Only the result pipeline sorts, and a query with hits needs no search index lookup.
+ assertEquals(counts + 1, stageCount("$count"));
+ assertEquals(sorts + 1, stageCount("$sort"));
+ assertEquals(indexLookups, stageCount("$listSearchIndexes"));
+ }
+
@Test
public void orderedValuesRemainDistinctBeyondTheMongotTokenLimit() throws Exception {
String query = "select [jcr:path] from [nt:unstructured] as s where "
@@ -187,6 +221,13 @@ public class MongotCoreQueryCompatibilityTest {
assertEquals(Set.copyOf(descending), Set.copyOf(ascending));
assertMonotonic(scores(repository.query(searchQuery + " order by [jcr:score] desc", "JCR-SQL2")), false);
assertMonotonic(scores(repository.query(searchQuery + " order by [jcr:score]", "JCR-SQL2")), true);
+
+ // $search already returns hits by descending score, so that order needs no $sort.
+ long searches = stageCount("$search");
+ long sorts = stageCount("$sort");
+ repository.paths(searchQuery + " order by [jcr:score] desc", "JCR-SQL2");
+ assertEquals(searches + 1, stageCount("$search"));
+ assertEquals(sorts, stageCount("$sort"));
}
@Test
@@ -260,6 +301,10 @@ public class MongotCoreQueryCompatibilityTest {
return scores;
}
+ private static long stageCount(String stage) {
+ return repository.stageCount(stage);
+ }
+
private static void assertMonotonic(List<Double> values, boolean ascending) {
for (int i = 1; i < values.size(); i++) {
assertTrue(values.toString(), ascending
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
index 158f285727..17a73b04d2 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
@@ -19,6 +19,7 @@ package org.apache.jackrabbit.oak.plugins.index.mongot.query;
import java.util.List;
import org.apache.jackrabbit.oak.api.Type;
+import org.apache.jackrabbit.oak.plugins.index.TestUtil;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotIndexDefinition;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotSearchConnectionRule;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotTestRepositoryBuilder;
@@ -34,6 +35,9 @@ import static com.mongodb.client.model.Filters.eq;
import static org.apache.jackrabbit.JcrConstants.JCR_PRIMARYTYPE;
import static org.apache.jackrabbit.oak.plugins.index.IndexConstants.REINDEX_PROPERTY_NAME;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
public class MongotReindexCompatibilityTest {
@@ -93,6 +97,23 @@ public class MongotReindexCompatibilityTest {
}
}
+ @Test
+ public void droppedSearchIndexFailsTheQueryInsteadOfReturningNoResults() throws Exception {
+ MongotTestRepositoryBuilder builder = titleBuilder();
+ content(builder.root(), "keep", "current-title");
+
+ try (MongotTestRepositoryBuilder.Fixture fixture = builder.build()) {
+ fixture.collection().dropSearchIndex(fixture.indexDefinition().getSearchIndexName());
+ TestUtil.assertEventually(() -> assertFalse(
+ fixture.collection().listSearchIndexes().iterator().hasNext()), 30_000);
+
+ // $search against a missing index returns no hits rather than an error.
+ IllegalStateException failure = assertThrows(IllegalStateException.class,
+ () -> fixture.paths(titleQuery("current-title"), "JCR-SQL2"));
+ assertTrue(failure.getMessage(), failure.getMessage().contains("is unavailable"));
+ }
+ }
+
private static MongotTestRepositoryBuilder titleBuilder() {
MongotTestRepositoryBuilder builder = new MongotTestRepositoryBuilder(mongo);
configureTitle(builder.definition());