You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by jo...@apache.org on 2026/09/29 11:16:56 UTC
(jackrabbit-oak) branch oak-search-mongot updated: OAK-search-mongot: return stored fields from Mongot instead of loading full documents (#3157)
This is an automated email from the ASF dual-hosted git repository.
Joscorbe pushed a commit to branch oak-search-mongot
in repository https://gitbox.apache.org/repos/asf/jackrabbit-oak.git
The following commit(s) were added to refs/heads/oak-search-mongot by this push:
new 8d88938c34 OAK-search-mongot: return stored fields from Mongot instead of loading full documents (#3157)
8d88938c34 is described below
commit 8d88938c3459b50783559e1d1d9b9324887c5e7d
Author: Ming He <me...@gmail.com>
AuthorDate: Tue Sep 29 04:16:50 2026 -0700
OAK-search-mongot: return stored fields from Mongot instead of loading full documents (#3157)
* OAK-search-mongot: return stored fields from Mongot instead of loading full documents
Every $search hit was followed by an implicit lookup of the full index document
on mongod before the path, type, property, sort and facet stages ran. Those
documents carry the analyzed and full-text fields, including extracted binary
text, so a query with many hits paid for loading all of that only to project
_path and the score.
With storedSource=true on the Oak index definition:
- the Search index stores just the fields the pipeline reads after $search,
and regular search queries set returnStoredSource so mongod skips the
lookup. Suggestions and spellchecks project fields that are not stored, so
they keep the lookup.
- the node-level full-text field is mapped with store:false unless a property
sets useInExcerpt. Mongot reads a hit's stored fields together, so keeping a
copy of that field made stored-source queries slower than lookups for large
documents. Without it Mongot cannot highlight the field, so rep:excerpt(.)
has no excerpt unless a property sets useInExcerpt, and the query no longer
requests that highlight.
The flag is read from the stored index definition, so it takes effect only
after a reindex builds a search index generation that stores those fields;
Mongot rejects returnStoredSource against an index built without them. The
include list is emitted in the sorted order Mongot reports back, otherwise
every indexing cycle would resubmit the definition and rebuild the index.
The core and advanced query compatibility tests now also run against a
stored-source index, and a new check asserts an indexing cycle leaves the
search index queryable in both modes.
* OAK-search-mongot: apply stored source only through a reindex
Mongot rejects returnStoredSource against a search index built without
storedSource, and highlighting against a full-text field it does not store.
Reading the storedSource property or the excerpt rules straight from the
live definition let a property edit change the query before the served
mapping changed, and made the next indexing cycle resubmit the definition,
which leaves the index unqueryable while mongot rebuilds it.
A reindex now publishes :storedSource and :fullTextStored with the new
collection seed. Queries and ordinary indexing cycles use the published
values; only the replacement generation is built from the current
definition.
_notNullProperties is no longer stored, since nothing reads it after
$search.
---------
Co-authored-by: José Andrés Cordero Benítez <Jo...@users.noreply.github.com>
---
oak-search-mongot/README.md | 4 +
.../index/mongot/MongotIndexDefinition.java | 49 ++++++++
.../index/mongot/index/MongoFieldNames.java | 6 +
.../index/mongot/index/MongotIndexWriter.java | 6 +-
.../index/MongotSearchIndexDefinitionBuilder.java | 18 +++
.../mongot/index/MongotSearchIndexManager.java | 7 +-
.../plugins/index/mongot/query/MongotIndex.java | 19 ++-
.../index/mongot/MongotIndexDefinitionTest.java | 27 +++++
.../MongotSearchIndexDefinitionBuilderTest.java | 43 +++++++
.../MongotAdvancedQueryCompatibilityTest.java | 10 +-
.../query/MongotCoreQueryCompatibilityTest.java | 25 +++-
.../query/MongotReindexCompatibilityTest.java | 132 +++++++++++++++++++++
...StoredSourceAdvancedQueryCompatibilityTest.java | 32 +++++
...ngotStoredSourceCoreQueryCompatibilityTest.java | 53 +++++++++
14 files changed, 420 insertions(+), 11 deletions(-)
diff --git a/oak-search-mongot/README.md b/oak-search-mongot/README.md
index c3676908a0..31957cbfd5 100644
--- a/oak-search-mongot/README.md
+++ b/oak-search-mongot/README.md
@@ -66,6 +66,10 @@ What is implemented
synthetic `$search` stage;
- lazy normal-result retrieval using a bounded Mongo aggregation cursor,
`queryFetchSizes`, and `queryTimeoutMs`;
+- opt-in `storedSource=true` (effective after reindex): the Search index stores
+ the fields read after `$search`, and queries return them instead of loading
+ full documents. The node-level full text is then kept for highlighting only
+ when a property sets `useInExcerpt`, so `rep:excerpt(.)` needs one;
- repository property updates and removals, exact and subtree deletion,
moves, aggregate refresh, and availability-preserving full reindex;
- Oak common-suite coverage for property indexes, full text, transformed path
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinition.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinition.java
index 93f2f53dd9..d7770e718e 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinition.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinition.java
@@ -21,6 +21,7 @@ import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Locale;
+import java.util.stream.StreamSupport;
import org.apache.jackrabbit.oak.api.Blob;
import org.apache.jackrabbit.oak.api.PropertyState;
@@ -37,6 +38,9 @@ public final class MongotIndexDefinition extends IndexDefinition {
public static final String PROP_COLLECTION_SEED = ":collectionSeed";
public static final String QUERY_FETCH_SIZES = "queryFetchSizes";
public static final String QUERY_TIMEOUT_MS = "queryTimeoutMs";
+ public static final String STORED_SOURCE = "storedSource";
+ public static final String PROP_SERVED_STORED_SOURCE = ":storedSource";
+ public static final String PROP_SERVED_FULL_TEXT_STORED = ":fullTextStored";
private static final int[] DEFAULT_QUERY_FETCH_SIZES = {10, 100, 1000};
private static final long DEFAULT_QUERY_TIMEOUT_MILLIS = 60_000L;
@@ -46,6 +50,10 @@ public final class MongotIndexDefinition extends IndexDefinition {
private final String searchIndexName;
private final int[] queryFetchSizes;
private final long queryTimeoutMillis;
+ private final boolean storedSourceRequested;
+ private final boolean storedSource;
+ private final boolean fullTextStoredRequested;
+ private final boolean fullTextStored;
public MongotIndexDefinition(NodeState root, NodeState definition, String indexPath) {
super(root, getIndexDefinitionState(definition), IndexFormatVersion.V2,
@@ -60,6 +68,18 @@ public final class MongotIndexDefinition extends IndexDefinition {
if (queryTimeoutMillis <= 0) {
throw new IllegalArgumentException(QUERY_TIMEOUT_MS + " must be positive");
}
+ this.storedSourceRequested = definition.getBoolean(STORED_SOURCE);
+ // Mongot rejects returnStoredSource against a search index built without storedSource,
+ // and highlighting against a full-text field it does not store. The served values change
+ // only when a reindex publishes a new collection generation.
+ this.storedSource = definition.getBoolean(PROP_SERVED_STORED_SOURCE);
+ // Mongot reads a hit's stored fields together, so a stored copy of the large full-text
+ // field makes stored-source reads slower than document lookups. Keep it only when
+ // excerpts need it for highlighting.
+ this.fullTextStoredRequested = !storedSourceRequested || hasExcerptProperties();
+ this.fullTextStored = definition.hasProperty(PROP_SERVED_FULL_TEXT_STORED)
+ ? definition.getBoolean(PROP_SERVED_FULL_TEXT_STORED)
+ : !storedSource;
}
public String getCollectionName() {
@@ -86,6 +106,35 @@ public final class MongotIndexDefinition extends IndexDefinition {
return queryTimeoutMillis;
}
+ /** Whether the served collection generation stores its post-search fields. */
+ public boolean isStoredSource() {
+ return isStoredSource(false);
+ }
+
+ /**
+ * @param nextGeneration {@code true} for the generation a reindex builds from the current
+ * definition, {@code false} for the served one
+ */
+ public boolean isStoredSource(boolean nextGeneration) {
+ return nextGeneration ? storedSourceRequested : storedSource;
+ }
+
+ /** Whether the served collection generation stores the full-text field for highlighting. */
+ public boolean isFullTextStored() {
+ return isFullTextStored(false);
+ }
+
+ public boolean isFullTextStored(boolean nextGeneration) {
+ return nextGeneration ? fullTextStoredRequested : fullTextStored;
+ }
+
+ private boolean hasExcerptProperties() {
+ return getDefinedRules().stream().anyMatch(rule ->
+ StreamSupport.stream(rule.getProperties().spliterator(), false)
+ .anyMatch(property -> property.stored)
+ || rule.getNamePatternsProperties().anyMatch(property -> property.stored));
+ }
+
public String getSynonymCollectionName() {
return MongotIndexNames.collectionName(getIndexPath()) + "_synonyms";
}
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongoFieldNames.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongoFieldNames.java
index 87129f148d..396d830f12 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongoFieldNames.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongoFieldNames.java
@@ -18,6 +18,7 @@ package org.apache.jackrabbit.oak.plugins.index.mongot.index;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
+import java.util.List;
public final class MongoFieldNames {
@@ -43,6 +44,11 @@ public final class MongoFieldNames {
public static final String DYNAMIC_BOOST_TOKENS = "dynamicBoostTokens";
public static final String DYNAMIC_BOOST_SCORES = "dynamicBoostScores";
+ // Every field the query pipeline reads on mongod after $search. Storing these on mongot
+ // lets queries skip loading full documents, whose full-text fields are the large part.
+ public static final List<String> POST_SEARCH_FIELDS = List.of(PATH, PARENT, ANCESTORS,
+ DEPTH, PRIMARY_TYPE, MIXIN_TYPES, NULL_PROPERTIES, TYPED, ORDERED, FACET);
+
private static final Base64.Encoder PROPERTY_ENCODER = Base64.getUrlEncoder().withoutPadding();
private MongoFieldNames() {
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotIndexWriter.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotIndexWriter.java
index 333876fb84..cb400458cc 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotIndexWriter.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotIndexWriter.java
@@ -101,7 +101,7 @@ public final class MongotIndexWriter implements FulltextIndexWriter<MongoDocumen
@Override
public boolean close(long timestamp) throws IOException {
if (updated) {
- searchIndexManager.ensureSearchIndex(definition, collectionName);
+ searchIndexManager.ensureSearchIndex(definition, collectionName, reindex);
if (reindex) {
searchIndexManager.awaitSearchIndexReady(definition, collectionName);
}
@@ -111,6 +111,10 @@ public final class MongotIndexWriter implements FulltextIndexWriter<MongoDocumen
if (reindex) {
definitionBuilder.setProperty(
MongotIndexDefinition.PROP_COLLECTION_SEED, collectionSeed);
+ definitionBuilder.setProperty(MongotIndexDefinition.PROP_SERVED_STORED_SOURCE,
+ definition.isStoredSource(true));
+ definitionBuilder.setProperty(MongotIndexDefinition.PROP_SERVED_FULL_TEXT_STORED,
+ definition.isFullTextStored(true));
}
}
return updated;
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilder.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilder.java
index 85ad5ff0d5..da9a12868b 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilder.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilder.java
@@ -69,6 +69,10 @@ final class MongotSearchIndexDefinitionBuilder {
}
static Document build(MongotIndexDefinition definition) {
+ return build(definition, false);
+ }
+
+ static Document build(MongotIndexDefinition definition, boolean nextGeneration) {
Analyzer analyzer = analyzer(definition.getDefinitionNodeState());
Document fields = new Document();
fields.append(MongoFieldNames.SYNC_TOKEN, new Document("type", "token"));
@@ -86,6 +90,14 @@ final class MongotSearchIndexDefinitionBuilder {
if (definition.isSpellcheckEnabled()) {
fields.append(MongoFieldNames.SPELLCHECK, new Document("type", "string"));
}
+ if (!definition.isFullTextStored(nextGeneration)) {
+ Document fullText = new Document("type", "string").append("store", false);
+ if (analyzer.name() != null) {
+ fullText.append("analyzer", analyzer.name())
+ .append("searchAnalyzer", analyzer.searchName());
+ }
+ fields.append(MongoFieldNames.FULLTEXT, fullText);
+ }
for (IndexDefinition.IndexingRule rule : definition.getDefinedRules()) {
for (PropertyDefinition property : rule.getSimilarityProperties()) {
fields.append(FieldNames.createSimilarityFieldName(
@@ -103,6 +115,12 @@ final class MongotSearchIndexDefinitionBuilder {
mappings.append("fields", fields);
}
Document result = new Document("mappings", mappings);
+ if (definition.isStoredSource(nextGeneration)) {
+ // Mongot reports the include list sorted. Emitting the same order keeps
+ // MongotSearchIndexManager from resubmitting, and so rebuilding, the index.
+ result.append("storedSource", new Document("include",
+ MongoFieldNames.POST_SEARCH_FIELDS.stream().sorted().toList()));
+ }
if (analyzer.name() != null) {
result.append("analyzer", analyzer.name())
.append("searchAnalyzer", analyzer.searchName());
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexManager.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexManager.java
index 9bd043b893..2a2cfc92eb 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexManager.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexManager.java
@@ -42,17 +42,18 @@ public final class MongotSearchIndexManager {
}
public void ensureSearchIndex(MongotIndexDefinition definition) {
- ensureSearchIndex(definition, definition.getCollectionName());
+ ensureSearchIndex(definition, definition.getCollectionName(), false);
}
- void ensureSearchIndex(MongotIndexDefinition definition, String collectionName) {
+ void ensureSearchIndex(MongotIndexDefinition definition, String collectionName,
+ boolean nextGeneration) {
if (!connection.getDatabase().listCollectionNames().into(new ArrayList<>())
.contains(collectionName)) {
connection.getDatabase().createCollection(collectionName);
}
ensureSynonyms(definition);
MongoCollection<Document> collection = connection.getDatabase().getCollection(collectionName);
- Document desiredDefinition = MongotSearchIndexDefinitionBuilder.build(definition);
+ Document desiredDefinition = MongotSearchIndexDefinitionBuilder.build(definition, nextGeneration);
for (Document index : collection.listSearchIndexes()) {
if (definition.getSearchIndexName().equals(index.getString("name"))) {
Document latestDefinition = index.get("latestDefinition", Document.class);
diff --git a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
index 1fc996b7fa..db422520e1 100644
--- a/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
+++ b/oak-search-mongot/src/main/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotIndex.java
@@ -406,13 +406,17 @@ final class MongotIndex extends FulltextIndex {
}
private static Document searchStage(MongotIndexDefinition definition, Document operator,
- List<String> highlightPaths) {
+ List<String> highlightPaths,
+ boolean returnStoredSource) {
Document search = new Document("index", definition.getSearchIndexName());
search.putAll(operator);
if (!highlightPaths.isEmpty()) {
Object path = highlightPaths.size() == 1 ? highlightPaths.get(0) : highlightPaths;
search.append("highlight", new Document("path", path));
}
+ if (returnStoredSource) {
+ search.append("returnStoredSource", true);
+ }
return new Document("$search", search);
}
@@ -441,8 +445,13 @@ final class MongotIndex extends FulltextIndex {
boolean hasSearch = hasFullText || hasSimilarity || hasNativeQuery;
if (hasSearch) {
Document searchOperator;
+ // Mongot can only highlight stored fields. Without a stored full text there is no
+ // node-level excerpt, as with Lucene for properties that are not useInExcerpt.
List<String> highlightPaths = MongotResultAdapter.highlightPaths(
- request.excerptColumns());
+ request.excerptColumns()).stream()
+ .filter(path -> definition.isFullTextStored()
+ || !MongoFieldNames.FULLTEXT.equals(path))
+ .toList();
List<Document> lexicalOperators = new ArrayList<>();
if (hasFullText) {
lexicalOperators.add(fullText.searchOperator());
@@ -462,7 +471,8 @@ final class MongotIndex extends FulltextIndex {
}
searchOperator = combineMust(operators);
}
- pipeline.add(searchStage(definition, searchOperator, highlightPaths));
+ pipeline.add(searchStage(definition, searchOperator, highlightPaths,
+ definition.isStoredSource()));
}
pipeline.addAll(filters.pipeline());
@@ -638,7 +648,8 @@ final class MongotIndex extends FulltextIndex {
.append(field, 1)
.append("_score", new Document("$meta", "searchScore"));
List<Document> pipeline = new ArrayList<>();
- pipeline.add(searchStage(definition, operator, List.of()));
+ // Suggestions and spellchecks project their own field, which is not stored on mongot.
+ pipeline.add(searchStage(definition, operator, List.of(), false));
MongotQueryTranslation filters = MongotQueryTranslator.translateFilter(
plan.getFilter(), getPlanResult(plan));
if (!filters.isSupported()) {
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinitionTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinitionTest.java
index adfe9e7d00..4573a3994d 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinitionTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/MongotIndexDefinitionTest.java
@@ -114,6 +114,33 @@ public class MongotIndexDefinitionTest {
assertEquals(1_234L, configured.getQueryTimeoutMillis());
}
+ @Test
+ public void servesStoredSourceOnlyOnceAReindexHasPublishedIt() {
+ MongotIndexDefinition defaults = newDefinition("defaults");
+ assertFalse(defaults.isStoredSource());
+ assertTrue(defaults.isFullTextStored());
+
+ NodeBuilder root = EmptyNodeState.EMPTY_NODE.builder();
+ NodeBuilder definitionBuilder = root.child(INDEX_DEFINITIONS_NAME).child("stored");
+ definitionBuilder.setProperty(JCR_PRIMARYTYPE, INDEX_DEFINITIONS_NODE_TYPE, Type.NAME);
+ definitionBuilder.setProperty(TYPE_PROPERTY_NAME, MongotIndexDefinition.TYPE_MONGOT);
+ definitionBuilder.setProperty(MongotIndexDefinition.STORED_SOURCE, true);
+
+ // The property alone is only a request; the served generation doesn't store fields yet.
+ MongotIndexDefinition requested = new MongotIndexDefinition(root.getNodeState(),
+ definitionBuilder.getNodeState(), "/oak:index/stored");
+ assertTrue(requested.isStoredSource(true));
+ assertFalse(requested.isStoredSource());
+ assertTrue(requested.isFullTextStored());
+
+ definitionBuilder.setProperty(MongotIndexDefinition.PROP_SERVED_STORED_SOURCE, true);
+ definitionBuilder.setProperty(MongotIndexDefinition.PROP_SERVED_FULL_TEXT_STORED, false);
+ MongotIndexDefinition served = new MongotIndexDefinition(root.getNodeState(),
+ definitionBuilder.getNodeState(), "/oak:index/stored");
+ assertTrue(served.isStoredSource());
+ assertFalse(served.isFullTextStored());
+ }
+
@Test
public void rejectsUnusableQueryBatchAndTimeoutProperties() {
assertThrows(IllegalArgumentException.class,
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilderTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilderTest.java
index 1c6e2b0b63..ca5dec2a52 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilderTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/index/MongotSearchIndexDefinitionBuilderTest.java
@@ -29,10 +29,47 @@ import org.junit.Test;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertThrows;
public class MongotSearchIndexDefinitionBuilderTest {
+ @Test
+ public void storesPostSearchFieldsOnlyWhenStoredSourceIsEnabled() {
+ assertNull(MongotSearchIndexDefinitionBuilder.build(definition(builder()), true)
+ .get("storedSource"));
+ // Until a reindex publishes it, the served generation keeps its original mapping.
+ assertNull(MongotSearchIndexDefinitionBuilder.build(storedSource(builder()), false)
+ .get("storedSource"));
+
+ Document searchDefinition = MongotSearchIndexDefinitionBuilder.build(
+ storedSource(builder()), true);
+
+ // The fields read after $search, in the sorted order mongot reports back.
+ assertEquals(new Document("include", List.of("_ancestors", "_depth", "_mixinTypes",
+ "_nullProperties", "_parent", "_path", "_primaryType", "facet", "ordered",
+ "typed")),
+ searchDefinition.get("storedSource"));
+ }
+
+ @Test
+ public void storesFullTextWithStoredSourceOnlyWhenExcerptsCanHighlightIt() {
+ IndexDefinitionBuilder withoutExcerpts = builder();
+ withoutExcerpts.indexRule("nt:base").property("title").analyzed().nodeScopeIndex();
+ Document fields = MongotSearchIndexDefinitionBuilder.build(storedSource(withoutExcerpts), true)
+ .get("mappings", Document.class).get("fields", Document.class);
+ assertEquals(new Document("type", "string").append("store", false)
+ .append("analyzer", "oak_default").append("searchAnalyzer", "lucene.standard"),
+ fields.get(MongoFieldNames.FULLTEXT));
+
+ IndexDefinitionBuilder withExcerpts = builder();
+ withExcerpts.indexRule("nt:base").property("title").analyzed().nodeScopeIndex()
+ .useInExcerpt();
+ assertNull(MongotSearchIndexDefinitionBuilder.build(storedSource(withExcerpts), true)
+ .get("mappings", Document.class).get("fields", Document.class)
+ .get(MongoFieldNames.FULLTEXT));
+ }
+
@Test
public void mapsOakDefaultAnalyzer() {
Document searchDefinition = MongotSearchIndexDefinitionBuilder.build(definition(builder()));
@@ -521,4 +558,10 @@ public class MongotSearchIndexDefinitionBuilderTest {
return new MongotIndexDefinition(EmptyNodeState.EMPTY_NODE, builder.build(),
"/oak:index/analyzer");
}
+
+ private static MongotIndexDefinition storedSource(IndexDefinitionBuilder builder) {
+ return new MongotIndexDefinition(EmptyNodeState.EMPTY_NODE, builder.build().builder()
+ .setProperty(MongotIndexDefinition.STORED_SOURCE, true).getNodeState(),
+ "/oak:index/analyzer");
+ }
}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotAdvancedQueryCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotAdvancedQueryCompatibilityTest.java
index 76b881ac03..7806476223 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotAdvancedQueryCompatibilityTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotAdvancedQueryCompatibilityTest.java
@@ -23,6 +23,7 @@ import java.util.Set;
import org.apache.jackrabbit.oak.api.Result;
import org.apache.jackrabbit.oak.api.ResultRow;
import org.apache.jackrabbit.oak.api.Type;
+import org.apache.jackrabbit.oak.plugins.index.mongot.MongotIndexDefinition;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotSearchConnectionRule;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotTestRepositoryBuilder;
import org.apache.jackrabbit.oak.plugins.index.search.FulltextIndexConstants;
@@ -43,12 +44,17 @@ public class MongotAdvancedQueryCompatibilityTest {
@ClassRule
public static final MongotSearchConnectionRule mongo = new MongotSearchConnectionRule();
- private static MongotTestRepositoryBuilder.Fixture repository;
+ static MongotTestRepositoryBuilder.Fixture repository;
@BeforeClass
public static void createRepository() throws Exception {
+ createRepository(false);
+ }
+
+ static void createRepository(boolean storedSource) throws Exception {
MongotTestRepositoryBuilder builder = new MongotTestRepositoryBuilder(mongo);
IndexDefinitionBuilder definition = builder.definition();
+ definition.getBuilderTree().setProperty(MongotIndexDefinition.STORED_SOURCE, storedSource);
configure(definition.indexRule("nt:base"));
configure(definition.indexRule("nt:unstructured"));
definition.aggregateRule("nt:unstructured", "jcr:content");
@@ -183,7 +189,7 @@ public class MongotAdvancedQueryCompatibilityTest {
assertTrue(plan, plan.contains("mongot:"));
}
- private static String plan(String query) throws Exception {
+ static String plan(String query) throws Exception {
return repository.query("explain " + query, "JCR-SQL2")
.getRows().iterator().next().getValue("plan").getValue(Type.STRING);
}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
index c9640799ed..eba1b6f368 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotCoreQueryCompatibilityTest.java
@@ -30,6 +30,7 @@ import org.apache.jackrabbit.oak.plugins.index.mongot.MongotSearchConnectionRule
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotTestRepositoryBuilder;
import org.apache.jackrabbit.oak.plugins.index.search.util.IndexDefinitionBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
+import org.bson.Document;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
@@ -38,6 +39,7 @@ import org.junit.Test;
import static org.apache.jackrabbit.JcrConstants.JCR_PRIMARYTYPE;
import static org.apache.jackrabbit.JcrConstants.JCR_MIXINTYPES;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
public class MongotCoreQueryCompatibilityTest {
@@ -45,12 +47,17 @@ public class MongotCoreQueryCompatibilityTest {
@ClassRule
public static final MongotSearchConnectionRule mongo = new MongotSearchConnectionRule();
- private static MongotTestRepositoryBuilder.Fixture repository;
+ static MongotTestRepositoryBuilder.Fixture repository;
@BeforeClass
public static void createRepository() throws Exception {
+ createRepository(false);
+ }
+
+ static void createRepository(boolean storedSource) throws Exception {
MongotTestRepositoryBuilder builder = new MongotTestRepositoryBuilder(mongo);
IndexDefinitionBuilder definition = builder.definition();
+ definition.getBuilderTree().setProperty(MongotIndexDefinition.STORED_SOURCE, storedSource);
definition.indexRule("nt:base").property("jcr:title")
.propertyIndex().analyzed().nodeScopeIndex();
IndexDefinitionBuilder.IndexRule rule = definition.indexRule("nt:unstructured");
@@ -191,6 +198,22 @@ public class MongotCoreQueryCompatibilityTest {
repository.paths(query, "JCR-SQL2", 5, 12));
}
+ @Test
+ public void indexingCycleKeepsTheSearchIndexQueryable() throws Exception {
+ Document before = repository.collection().listSearchIndexes().first();
+ assertNotNull(before.toJson(), before.get("latestVersion"));
+
+ repository.mutate(root -> root.child("unrelated")
+ .setProperty(JCR_PRIMARYTYPE, "nt:unstructured", Type.NAME)
+ .setProperty("jcr:title", "zebra"));
+ repository.index();
+
+ // Resubmitting an unchanged definition makes mongot rebuild, and stop serving, the index.
+ Document after = repository.collection().listSearchIndexes().first();
+ assertEquals(after.toJson(), "READY", after.getString("status"));
+ assertEquals(before.get("latestVersion"), after.get("latestVersion"));
+ }
+
@Test
public void propertyOnlyQueryUsesMongotIndexWithoutSearchMetadata() throws Exception {
String query = "select [jcr:path] from [nt:unstructured] as s where "
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
index 17a73b04d2..b406b134ea 100644
--- a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotReindexCompatibilityTest.java
@@ -18,6 +18,7 @@ package org.apache.jackrabbit.oak.plugins.index.mongot.query;
import java.util.List;
+import org.apache.jackrabbit.oak.api.PropertyValue;
import org.apache.jackrabbit.oak.api.Type;
import org.apache.jackrabbit.oak.plugins.index.TestUtil;
import org.apache.jackrabbit.oak.plugins.index.mongot.MongotIndexDefinition;
@@ -28,14 +29,18 @@ import org.apache.jackrabbit.oak.plugins.index.mongot.index.MongoFieldNames;
import org.apache.jackrabbit.oak.plugins.index.search.util.IndexDefinitionBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeBuilder;
import org.apache.jackrabbit.oak.spi.state.NodeState;
+import org.bson.Document;
import org.junit.ClassRule;
import org.junit.Test;
import static com.mongodb.client.model.Filters.eq;
import static org.apache.jackrabbit.JcrConstants.JCR_PRIMARYTYPE;
import static org.apache.jackrabbit.oak.plugins.index.IndexConstants.REINDEX_PROPERTY_NAME;
+import static org.apache.jackrabbit.oak.plugins.index.search.FulltextIndexConstants.PROP_USE_IN_EXCERPT;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;
@@ -98,6 +103,133 @@ public class MongotReindexCompatibilityTest {
}
@Test
+ public void storedSourceTakesEffectOnlyWhenAReindexPublishesIt() throws Exception {
+ MongotTestRepositoryBuilder builder = titleBuilder();
+ content(builder.root(), "keep", "current-title");
+ String query = titleQuery("current-title") + " order by [jcr:path]";
+
+ try (MongotTestRepositoryBuilder.Fixture fixture = builder.build()) {
+ fixture.mutate(root -> {
+ root.getChildNode("oak:index").getChildNode("mongoSearch")
+ .setProperty(MongotIndexDefinition.STORED_SOURCE, true);
+ content(root, "added", "current-title");
+ });
+ fixture.index();
+
+ // An ordinary indexing cycle keeps serving the generation built without stored fields.
+ assertFalse(servedDefinition(fixture).containsKey("storedSource"));
+ assertFalse(plan(fixture, query).contains("returnStoredSource"));
+ fixture.awaitPaths(query, "JCR-SQL2", List.of("/content/added", "/content/keep"));
+
+ fixture.mutate(root -> root.getChildNode("oak:index").getChildNode("mongoSearch")
+ .setProperty(REINDEX_PROPERTY_NAME, true));
+ fixture.index();
+
+ assertTrue(servedDefinition(fixture).containsKey("storedSource"));
+ assertTrue(plan(fixture, query).contains("returnStoredSource=true"));
+ fixture.awaitPaths(query, "JCR-SQL2", List.of("/content/added", "/content/keep"));
+ }
+ }
+
+ @Test
+ public void fullTextStorageFollowsThePublishedGenerationWhenExcerptRulesChange()
+ throws Exception {
+ MongotTestRepositoryBuilder builder = titleBuilder();
+ content(builder.root(), "keep", "current-title");
+ String query = "select [jcr:path], [rep:excerpt(.)] from [nt:unstructured] as s "
+ + "where contains(s.[jcr:title], 'current-title') order by [jcr:path]";
+
+ try (MongotTestRepositoryBuilder.Fixture fixture = builder.build()) {
+ fixture.mutate(root -> mongoSearch(root)
+ .setProperty(MongotIndexDefinition.STORED_SOURCE, true)
+ .setProperty(REINDEX_PROPERTY_NAME, true));
+ fixture.index();
+ assertNotNull(fullTextMapping(fixture));
+
+ // Mongot rejects highlighting an unstored field, and resubmitting the mapping makes
+ // it rebuild the index, so new excerpt rules wait for the next reindex.
+ Object version = latestVersion(fixture);
+ fixture.mutate(root -> {
+ titleRule(root).setProperty(PROP_USE_IN_EXCERPT, true);
+ content(root, "added", "current-title");
+ });
+ fixture.index();
+ assertEquals(version, latestVersion(fixture));
+ assertNotNull(fullTextMapping(fixture));
+ fixture.awaitPaths(query, "JCR-SQL2", List.of("/content/added", "/content/keep"));
+ assertNull(excerpt(fixture, query));
+
+ fixture.mutate(root -> mongoSearch(root).setProperty(REINDEX_PROPERTY_NAME, true));
+ fixture.index();
+ assertNull(fullTextMapping(fixture));
+ fixture.awaitPaths(query, "JCR-SQL2", List.of("/content/added", "/content/keep"));
+ assertTrue(excerpt(fixture, query), excerpt(fixture, query).contains("<strong>"));
+
+ // Removing the rule keeps the served full-text copy until the next reindex too.
+ version = latestVersion(fixture);
+ fixture.mutate(root -> {
+ titleRule(root).removeProperty(PROP_USE_IN_EXCERPT);
+ content(root, "later", "current-title");
+ });
+ fixture.index();
+ assertEquals(version, latestVersion(fixture));
+ assertNull(fullTextMapping(fixture));
+ fixture.awaitPaths(query, "JCR-SQL2",
+ List.of("/content/added", "/content/keep", "/content/later"));
+
+ fixture.mutate(root -> mongoSearch(root).setProperty(REINDEX_PROPERTY_NAME, true));
+ fixture.index();
+ assertNotNull(fullTextMapping(fixture));
+ fixture.awaitPaths(query, "JCR-SQL2",
+ List.of("/content/added", "/content/keep", "/content/later"));
+ }
+ }
+
+ private static NodeBuilder mongoSearch(NodeBuilder root) {
+ return root.getChildNode("oak:index").getChildNode("mongoSearch");
+ }
+
+ private static NodeBuilder titleRule(NodeBuilder root) {
+ NodeBuilder properties = mongoSearch(root).getChildNode("indexRules")
+ .getChildNode("nt:unstructured").getChildNode("properties");
+ for (String name : properties.getChildNodeNames()) {
+ if ("jcr:title".equals(properties.getChildNode(name).getString("name"))) {
+ return properties.getChildNode(name);
+ }
+ }
+ throw new AssertionError("no jcr:title property rule");
+ }
+
+ private static Document fullTextMapping(MongotTestRepositoryBuilder.Fixture fixture) {
+ return servedDefinition(fixture).get("mappings", Document.class)
+ .get("fields", Document.class).get(MongoFieldNames.FULLTEXT, Document.class);
+ }
+
+ private static Object latestVersion(MongotTestRepositoryBuilder.Fixture fixture) {
+ Document index = fixture.collection().listSearchIndexes().first();
+ assertEquals(index.toJson(), "READY", index.getString("status"));
+ assertNotNull(index.toJson(), index.get("latestVersion"));
+ return index.get("latestVersion");
+ }
+
+ private static String excerpt(MongotTestRepositoryBuilder.Fixture fixture, String query)
+ throws Exception {
+ PropertyValue excerpt = fixture.query(query, "JCR-SQL2").getRows().iterator().next()
+ .getValue("rep:excerpt(.)");
+ return excerpt == null ? null : excerpt.getValue(Type.STRING);
+ }
+
+ private static Document servedDefinition(MongotTestRepositoryBuilder.Fixture fixture) {
+ return fixture.collection().listSearchIndexes().first()
+ .get("latestDefinition", Document.class);
+ }
+
+ private static String plan(MongotTestRepositoryBuilder.Fixture fixture, String query)
+ throws Exception {
+ return fixture.query("explain " + query, "JCR-SQL2").getRows().iterator().next()
+ .getValue("plan").getValue(Type.STRING);
+ }
+
public void droppedSearchIndexFailsTheQueryInsteadOfReturningNoResults() throws Exception {
MongotTestRepositoryBuilder builder = titleBuilder();
content(builder.root(), "keep", "current-title");
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceAdvancedQueryCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceAdvancedQueryCompatibilityTest.java
new file mode 100644
index 0000000000..01d88af529
--- /dev/null
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceAdvancedQueryCompatibilityTest.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.index.mongot.query;
+
+import org.junit.BeforeClass;
+
+/**
+ * Runs the advanced query contracts, including facets, excerpts, suggestions and
+ * spellchecks, against a search index that returns stored fields.
+ */
+public class MongotStoredSourceAdvancedQueryCompatibilityTest
+ extends MongotAdvancedQueryCompatibilityTest {
+
+ @BeforeClass
+ public static void createRepository() throws Exception {
+ createRepository(true);
+ }
+}
diff --git a/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceCoreQueryCompatibilityTest.java b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceCoreQueryCompatibilityTest.java
new file mode 100644
index 0000000000..9b3c496201
--- /dev/null
+++ b/oak-search-mongot/src/test/java/org/apache/jackrabbit/oak/plugins/index/mongot/query/MongotStoredSourceCoreQueryCompatibilityTest.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.index.mongot.query;
+
+import java.util.List;
+
+import org.apache.jackrabbit.oak.api.Type;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Runs the core query contracts against a search index that returns stored fields, so
+ * every filter, path, type, null-check and ordering stage must work without full documents.
+ */
+public class MongotStoredSourceCoreQueryCompatibilityTest extends MongotCoreQueryCompatibilityTest {
+
+ @BeforeClass
+ public static void createRepository() throws Exception {
+ createRepository(true);
+ }
+
+ @Test
+ public void excerptQueryReadsStoredFieldsWithoutHighlightingUnstoredFullText() throws Exception {
+ String query = "select [jcr:path], [rep:excerpt(.)] from [nt:unstructured] as s where "
+ + "contains(s.*, 'archived') and isdescendantnode(s, '/content/site')";
+ String plan = repository.query("explain " + query, "JCR-SQL2")
+ .getRows().iterator().next().getValue("plan").getValue(Type.STRING);
+ assertTrue(plan, plan.contains("returnStoredSource=true"));
+ // No property sets useInExcerpt, so the full text is not stored and mongot would
+ // reject a highlight request for it.
+ assertFalse(plan, plan.contains("highlight="));
+
+ assertEquals(List.of("/content/site/b"), repository.paths(query, "JCR-SQL2"));
+ }
+}