You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by th...@apache.org on 2026/04/09 11:15:52 UTC
(jackrabbit-oak) branch OAK-12117 updated: OAK-12117 - Limit the maximum number of indexed tags per document (#2789)
This is an automated email from the ASF dual-hosted git repository.
thomasm pushed a commit to branch OAK-12117
in repository https://gitbox.apache.org/repos/asf/jackrabbit-oak.git
The following commit(s) were added to refs/heads/OAK-12117 by this push:
new 2790c16532 OAK-12117 - Limit the maximum number of indexed tags per document (#2789)
2790c16532 is described below
commit 2790c16532d9df8ec113c510f7a568bbdc837b86
Author: Anton Hosgood <ah...@icloud.com>
AuthorDate: Thu Apr 9 13:15:47 2026 +0200
OAK-12117 - Limit the maximum number of indexed tags per document (#2789)
* feat: limit max tag counts
* update docs
* fix: logging
* docs: update canonical index definition
* refactor: collectDynamicBoost returns void
* refactor: add DocumentBuildContext static class
* feat: add per-tag error handling in indexTopDynamicBoost
* refactor: rename collected dynamic boost tags parameter
---------
Co-authored-by: Anton Hosgood <ah...@adobe.com>
---
oak-doc/src/site/markdown/query/lucene.md | 27 ++++++
.../index/lucene/LuceneDocumentMakerTest.java | 41 +++++++++
.../dynamicBoost/LuceneDynamicBoostTest.java | 36 +++++++-
.../index/search/FulltextIndexConstants.java | 12 +++
.../oak/plugins/index/search/IndexDefinition.java | 26 ++++++
.../search/spi/editor/FulltextDocumentMaker.java | 98 +++++++++++++++++-----
.../oak/plugins/index/DynamicBoostCommonTest.java | 10 ++-
.../plugins/index/search/IndexDefinitionTest.java | 26 ++++++
8 files changed, 250 insertions(+), 26 deletions(-)
diff --git a/oak-doc/src/site/markdown/query/lucene.md b/oak-doc/src/site/markdown/query/lucene.md
index a41649ec3e..048d0853ec 100644
--- a/oak-doc/src/site/markdown/query/lucene.md
+++ b/oak-doc/src/site/markdown/query/lucene.md
@@ -155,6 +155,8 @@ Below is the canonical index definition structure
- excludedPaths (string) multiple
- maxFieldLength (long) = 10000
- maxTagLength (long) = 100
+ - maxSimilarityTagsCount (long) = 50
+ - maxDynamicBoostCount (long) = 50
- refresh (boolean)
- useIfExists (string)
- blobSize (long) = 32768
@@ -241,6 +243,20 @@ selectionPolicy
Set to -1 to disable the length check entirely.
See [Dynamic Boost](#dynamic-boost) and [Search by similar feature vectors](#similar-fv) for details.
+[maxSimilarityTagsCount][OAK-12117]
+: Optional integer property. Defaults to 50.
+: Maximum number of similarity tags to index per document.
+ When the limit is exceeded, only the first N tags (in order of appearance) are indexed and subsequent tags are skipped.
+ Set to -1 to disable the limit entirely.
+ See [Search by similar feature vectors](#similar-fv) for details.
+
+[maxDynamicBoostCount][OAK-12117]
+: Optional integer property. Defaults to 50.
+: Maximum number of dynamic boost tags to index per document.
+ When the limit is exceeded, tags are sorted by confidence (descending) and only the top N are indexed.
+ Set to -1 to disable the limit entirely.
+ See [Dynamic Boost](#dynamic-boost) for details.
+
refresh
: Optional boolean property.
: Used to refresh the stored index definition. See [Effective Index Definition](#stored-index-definition)
@@ -1244,6 +1260,11 @@ This prevents unexpectedly long values from being indexed as dynamic boost tags.
The limit can be changed by setting the `maxTagLength` property on the index definition,
or disabled entirely by setting it to -1. See [OAK-12101][OAK-12101].
+Additionally, the maximum number of dynamic boost tags indexed per document can be controlled
+with the `maxDynamicBoostCount` property (default 50). When the number of collected tags exceeds this limit,
+tags are sorted by confidence in descending order and only the top N are indexed.
+Set to -1 to disable the limit entirely. See [OAK-12117][OAK-12117].
+
### <a name="native-query"></a>Native Query and Index Selection
`@deprecated Oak 1.46`
@@ -1720,6 +1741,11 @@ This prevents unexpectedly long values from being indexed as similarity tags.
The limit can be changed by setting the `maxTagLength` property on the index definition,
or disabled entirely by setting it to -1. See [OAK-12101][OAK-12101].
+Additionally, the maximum number of similarity tags indexed per document can be controlled
+with the `maxSimilarityTagsCount` property (default 50). When the number of tags exceeds this limit,
+only the first N tags (in order of appearance) are kept and subsequent tags are skipped.
+Set to -1 to disable the limit entirely. See [OAK-12117][OAK-12117].
+
See also [OAK-7575](https://issues.apache.org/jira/browse/OAK-7575).
@@ -2250,6 +2276,7 @@ SELECT rep:facet(title) FROM [app:Asset] WHERE [title] IS NOT NULL
[OAK-8971]: https://issues.apache.org/jira/browse/OAK-8971
[OAK-9625]: https://issues.apache.org/jira/browse/OAK-9625
[OAK-12101]: https://issues.apache.org/jira/browse/OAK-12101
+[OAK-12117]: https://issues.apache.org/jira/browse/OAK-12117
[luke]: https://code.google.com/p/luke/
[tika]: http://tika.apache.org/
[oak-console]: https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#console
diff --git a/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneDocumentMakerTest.java b/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneDocumentMakerTest.java
index 2c2e804fbe..9b6f9d44ec 100644
--- a/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneDocumentMakerTest.java
+++ b/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneDocumentMakerTest.java
@@ -102,4 +102,45 @@ public class LuceneDocumentMakerTest {
assertNull(doc.get(FieldNames.SIMILARITY_TAGS));
}
+ @Test
+ public void similarityTagCountLimit() throws Exception{
+ LuceneIndexDefinitionBuilder builder = new LuceneIndexDefinitionBuilder();
+ builder.indexRule("nt:base")
+ .property("jcr:primaryType")
+ .propertyIndex();
+ builder.indexRule("nt:base")
+ .property("tag1")
+ .similarityTags(true);
+ builder.indexRule("nt:base")
+ .property("tag2")
+ .similarityTags(true);
+ builder.indexRule("nt:base")
+ .property("tag3")
+ .similarityTags(true);
+ builder.indexRule("nt:base")
+ .property("tag4")
+ .similarityTags(true);
+ builder.indexRule("nt:base")
+ .property("tag5")
+ .similarityTags(true);
+
+ builder.getBuilderTree().setProperty(FulltextIndexConstants.MAX_SIMILARITY_TAGS_COUNT, 3);
+
+ LuceneIndexDefinition defn = LuceneIndexDefinition.newLuceneBuilder(root, builder.build(), "/foo").build();
+ LuceneDocumentMaker docMaker = new LuceneDocumentMaker(defn,
+ defn.getApplicableIndexingRule("nt:base"), "/x");
+
+ NodeBuilder test = EMPTY_NODE.builder();
+ test.setProperty("tag1", "value1");
+ test.setProperty("tag2", "value2");
+ test.setProperty("tag3", "value3");
+ test.setProperty("tag4", "value4");
+ test.setProperty("tag5", "value5");
+ Document doc = docMaker.makeDocument(test.getNodeState());
+ assertNotNull(doc);
+
+ String[] tags = doc.getValues(FieldNames.SIMILARITY_TAGS);
+ assertEquals(3, tags.length);
+ }
+
}
diff --git a/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/dynamicBoost/LuceneDynamicBoostTest.java b/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/dynamicBoost/LuceneDynamicBoostTest.java
index f8c17f6e6b..a37c7b0146 100644
--- a/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/dynamicBoost/LuceneDynamicBoostTest.java
+++ b/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/dynamicBoost/LuceneDynamicBoostTest.java
@@ -186,7 +186,7 @@ public class LuceneDynamicBoostTest extends DynamicBoostCommonTest {
@Test
public void dynamicBoostMaxLengthFiltering() throws Exception {
- createAssetsIndexAndProperties(false, false, true, 10);
+ createAssetsIndexAndProperties(false, false, true, 10, 10);
Tree testParent = createNodeWithType(root.getTree("/"), "test", JcrConstants.NT_UNSTRUCTURED, "");
@@ -211,10 +211,40 @@ public class LuceneDynamicBoostTest extends DynamicBoostCommonTest {
});
}
+ @Test
+ public void dynamicBoostCountLimit() throws Exception {
+ createAssetsIndexAndProperties(false, false, true, 10, 2);
+
+ Tree testParent = createNodeWithType(root.getTree("/"), "test", JcrConstants.NT_UNSTRUCTURED, "");
+
+ Tree predicted1 = createAssetNodeWithPredicted(testParent, "asset1", "test");
+ createPredictedTag(predicted1, "lowconf1", 0.1);
+ createPredictedTag(predicted1, "lowconf2", 0.2);
+ createPredictedTag(predicted1, "highconf1", 0.9);
+ createPredictedTag(predicted1, "highconf2", 0.8);
+
+ Tree predicted2 = createAssetNodeWithPredicted(testParent, "asset2", "test");
+ createPredictedTag(predicted2, "highconf1", 0.9);
+ createPredictedTag(predicted2, "highconf2", 0.8);
+
+ root.commit();
+
+ assertEventually(() -> {
+ assertQuery("//element(*, dam:Asset)[jcr:contains(., 'highconf1')]", XPATH,
+ List.of("/test/asset1", "/test/asset2"));
+ assertQuery("//element(*, dam:Asset)[jcr:contains(., 'highconf2')]", XPATH,
+ List.of("/test/asset1", "/test/asset2"));
+
+ assertQuery("//element(*, dam:Asset)[jcr:contains(., 'lowconf1')]", XPATH, List.of());
+ assertQuery("//element(*, dam:Asset)[jcr:contains(., 'lowconf2')]", XPATH, List.of());
+ });
+ }
+
@Override
- protected void createAssetsIndexAndProperties(boolean lite, boolean similarityTags, boolean useInFullTextQuery, Integer maxTagLength) throws Exception {
+ protected void createAssetsIndexAndProperties(boolean lite, boolean similarityTags, boolean useInFullTextQuery,
+ Integer maxTagLength, Integer maxTagCount) throws Exception {
factory.queryTermsProvider = new FulltextQueryTermsProviderImpl();
- super.createAssetsIndexAndProperties(lite, similarityTags, useInFullTextQuery, maxTagLength);
+ super.createAssetsIndexAndProperties(lite, similarityTags, useInFullTextQuery, maxTagLength, maxTagCount);
}
private String runIndexingTest(Class<?> loggerClass, boolean nameProperty) throws CommitFailedException {
diff --git a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/FulltextIndexConstants.java b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/FulltextIndexConstants.java
index aa29ddad5d..bad8025778 100644
--- a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/FulltextIndexConstants.java
+++ b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/FulltextIndexConstants.java
@@ -257,6 +257,18 @@ public interface FulltextIndexConstants {
*/
String MAX_TAG_LENGTH = "maxTagLength";
+ /**
+ * Maximum number of similarity tags to index per document.
+ * Set to -1 to disable the limit
+ */
+ String MAX_SIMILARITY_TAGS_COUNT = "maxSimilarityTagsCount";
+
+ /**
+ * Maximum number of dynamic boost tags to index per document. The top N tags by confidence are indexed.
+ * Set to -1 to disable the limit
+ */
+ String MAX_DYNAMIC_BOOST_COUNT = "maxDynamicBoostCount";
+
/**
* whether use this property values for suggestions
*/
diff --git a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinition.java b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinition.java
index 0798a2485c..0459375b9a 100644
--- a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinition.java
+++ b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinition.java
@@ -151,6 +151,16 @@ public class IndexDefinition implements Aggregate.AggregateMapper {
*/
public static final int DEFAULT_MAX_TAG_LENGTH = 100;
+ /**
+ * Default maximum number of similarity tags indexed per document.
+ */
+ public static final int DEFAULT_MAX_SIMILARITY_TAGS_COUNT = 50;
+
+ /**
+ * Default maximum number of dynamic boost tags indexed per document.
+ */
+ public static final int DEFAULT_MAX_DYNAMIC_BOOST_COUNT = 50;
+
public static final int DEFAULT_MAX_EXTRACT_LENGTH = -10;
/**
@@ -281,6 +291,10 @@ public class IndexDefinition implements Aggregate.AggregateMapper {
private final int maxTagLength;
+ private final int maxSimilarityTagsCount;
+
+ private final int maxDynamicBoostCount;
+
private final int maxExtractLength;
private final int suggesterUpdateFrequencyMinutes;
@@ -478,6 +492,10 @@ public class IndexDefinition implements Aggregate.AggregateMapper {
this.maxFieldLength = getOptionalValue(defn, FulltextIndexConstants.MAX_FIELD_LENGTH, DEFAULT_MAX_FIELD_LENGTH);
this.maxTagLength = getOptionalValue(defn, FulltextIndexConstants.MAX_TAG_LENGTH, DEFAULT_MAX_TAG_LENGTH);
+ this.maxSimilarityTagsCount = getOptionalValue(defn,
+ FulltextIndexConstants.MAX_SIMILARITY_TAGS_COUNT, DEFAULT_MAX_SIMILARITY_TAGS_COUNT);
+ this.maxDynamicBoostCount = getOptionalValue(defn,
+ FulltextIndexConstants.MAX_DYNAMIC_BOOST_COUNT, DEFAULT_MAX_DYNAMIC_BOOST_COUNT);
this.costPerEntry = getOptionalValue(defn, FulltextIndexConstants.COST_PER_ENTRY, getDefaultCostPerEntry(version));
this.costPerExecution = getOptionalValue(defn, FulltextIndexConstants.COST_PER_EXECUTION, 1.0);
this.hasCustomTikaConfig = getTikaConfigNode().exists();
@@ -702,6 +720,14 @@ public class IndexDefinition implements Aggregate.AggregateMapper {
return maxTagLength;
}
+ public int getMaxSimilarityTagsCount() {
+ return maxSimilarityTagsCount;
+ }
+
+ public int getMaxDynamicBoostCount() {
+ return maxDynamicBoostCount;
+ }
+
public int getMaxExtractLength() {
return maxExtractLength;
}
diff --git a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/spi/editor/FulltextDocumentMaker.java b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/spi/editor/FulltextDocumentMaker.java
index ee3f676826..bea8b77e62 100644
--- a/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/spi/editor/FulltextDocumentMaker.java
+++ b/oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/spi/editor/FulltextDocumentMaker.java
@@ -40,7 +40,9 @@ import org.slf4j.LoggerFactory;
import javax.jcr.PropertyType;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -128,7 +130,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
* @param nodeName the current node name
* @param value the value to be indexed
* @param confidence the confidence (or weight) used for re-scoring
- * @return {@code true} id the value has been added, otherwise {@code false}
+ * @return {@code true} if the value has been added, otherwise {@code false}
*/
protected abstract boolean indexDynamicBoost(D doc, String parent, String nodeName, String value, double confidence);
@@ -169,6 +171,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
boolean facet = false;
D document = initDoc();
+ DocumentBuildContext ctx = new DocumentBuildContext();
boolean dirty = false;
// we make a copy of the modified properties names. These will be removed while iterating over all properties.
@@ -197,7 +200,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
dirty |= addTypedOrderedFields(document, property, pname, pd);
}
- var indexed = indexProperty(path, document, state, property, pname, pd);
+ var indexed = indexProperty(path, document, ctx, state, property, pname, pd);
if (indexed) {
dirty = true;
// property was indexed, so remove from the removed list
@@ -207,12 +210,19 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
facet |= pd.facet;
}
- ResultCollector resultCollector = indexAggregates(path, document, state);
+ ResultCollector resultCollector = indexAggregates(path, document, ctx, state);
dirty |= resultCollector.dirtyFlag; // any (aggregate) indexing happened
facet |= resultCollector.facetFlag; // facet indexing during (index-time) aggregation
dirty |= indexNullCheckEnabledProps(path, document, state);
dirty |= indexFunctionRestrictions(path, document, state);
dirty |= indexNotNullCheckEnabledProps(path, document, state);
+ int dynamicBoostTagCount = ctx.collectedBoosts.size();
+ int maxDynamicBoostCount = definition.getMaxDynamicBoostCount();
+ if (maxDynamicBoostCount >= 0 && dynamicBoostTagCount > maxDynamicBoostCount) {
+ log.warn("[{}] Number of collected dynamic boost tags ({}) exceeds the maximum allowed ({}). Some tags will be skipped",
+ getIndexName(), dynamicBoostTagCount, maxDynamicBoostCount);
+ }
+ dirty |= indexTopDynamicBoost(document, ctx.collectedBoosts, maxDynamicBoostCount);
dirty |= augmentCustomFields(path, document, state);
if (!propertiesToRemove.isEmpty()) {
@@ -266,6 +276,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
private boolean indexProperty(String path,
D doc,
+ DocumentBuildContext ctx,
NodeState state,
PropertyState property,
String pname,
@@ -295,9 +306,9 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
}
if (!definition.isDynamicBoostLiteEnabled() && pd.dynamicBoost) {
try {
- dirty |= indexDynamicBoost(doc, pname, pd.nodeName, state);
+ collectDynamicBoost(pname, pd.nodeName, state, ctx.collectedBoosts);
} catch (Exception e) {
- log.error("Could not index dynamic boost for property {} and definition {}", property, pd, e);
+ log.error("Could not collect dynamic boost for property {} and definition {}", property, pd, e);
}
}
@@ -348,11 +359,15 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
}
if (pd.similarityTags) {
String value = property.getValue(Type.STRING);
- if (isTagWithinLengthLimit(value)) {
- dirty |= indexSimilarityTag(doc, value);
+ if (isSimilarityTagWithinLimits(value, ctx.similarityTagCount)) {
+ if (indexSimilarityTag(doc, value)) {
+ ctx.similarityTagCount++;
+ dirty = true;
+ }
} else if (!LOG_SILENCER.silence(pname)) {
- log.warn("[{}] Skipping similarity tag for property {}. Value length {} exceeds maximum allowed length",
- getIndexName(), pname, value.length());
+ log.warn("[{}] Skipping similarity tag for property {}. Either value length {} exceeds maximum" +
+ "allowed length or number of indexed tags {} exceeds maximum allowed count",
+ getIndexName(), pname, value.length(), ctx.similarityTagCount);
}
}
@@ -592,8 +607,8 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
/*
* index aggregates on a certain path
*/
- private ResultCollector indexAggregates(final String path, final D document, final NodeState state) {
- ResultCollector resultCollector = new ResultCollector(path, document, state);
+ private ResultCollector indexAggregates(final String path, final D document, final DocumentBuildContext ctx, final NodeState state) {
+ ResultCollector resultCollector = new ResultCollector(path, document, ctx, state);
indexingRule.getAggregate().collectAggregates(state, resultCollector);
return resultCollector;
}
@@ -601,13 +616,15 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
private class ResultCollector implements Aggregate.ResultCollector {
private final String path;
private final D document;
+ private final DocumentBuildContext ctx;
private final NodeState state;
private boolean dirtyFlag = false;
private boolean facetFlag = false;
- ResultCollector(String path, D document, NodeState state) {
+ ResultCollector(String path, D document, DocumentBuildContext ctx, NodeState state) {
this.path = path;
this.document = document;
+ this.ctx = ctx;
this.state = state;
}
@@ -621,7 +638,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
if (result.pd.ordered) {
dirtyFlag |= addTypedOrderedFields(document, result.propertyState, result.propertyPath, result.pd);
}
- dirtyFlag |= indexProperty(path, document, state, result.propertyState, result.propertyPath, result.pd);
+ dirtyFlag |= indexProperty(path, document, ctx, state, result.propertyState, result.propertyPath, result.pd);
if (result.pd.facet) {
facetFlag = true;
@@ -695,13 +712,16 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
return dirty;
}
- protected boolean indexDynamicBoost(D doc, String propertyName, String nodeName, NodeState nodeState) {
+ /**
+ * Collects dynamic boost tags from a NodeState property into {@code collectedBoosts} for later indexing.
+ */
+ private void collectDynamicBoost(String propertyName, String nodeName, NodeState nodeState, List<DynamicBoost> collectedBoosts) {
NodeState propertyNode = nodeState;
String parentName = PathUtils.getParentPath(propertyName);
for (String c : PathUtils.elements(parentName)) {
propertyNode = propertyNode.getChildNode(c);
}
- boolean added = false;
+
for (String childNodeName : propertyNode.getChildNodeNames()) {
NodeState dynaTag = propertyNode.getChildNode(childNodeName);
PropertyState p = dynaTag.getProperty(DYNAMIC_BOOST_TAG_NAME);
@@ -714,7 +734,7 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
continue;
}
String dynaTagValue = p.getValue(Type.STRING);
- if (!isTagWithinLengthLimit(dynaTagValue)) {
+ if (!isTagLengthWithinLimits(dynaTagValue)) {
if (!LOG_SILENCER.silence(p.getName())) {
log.warn("[{}] Skipping dynamic boost tag for property {}. Value length {} exceeds maximum allowed length",
getIndexName(), p.getName(), dynaTagValue.length());
@@ -742,22 +762,50 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
continue;
}
- if (indexDynamicBoost(doc, parentName, nodeName, dynaTagValue, dynaTagConfidence)) {
- added = true;
- }
+ collectedBoosts.add(new DynamicBoost(parentName, nodeName, dynaTagValue, dynaTagConfidence));
}
- return added;
}
protected String getIndexName() {
return definition.getIndexName();
}
- private boolean isTagWithinLengthLimit(String value) {
+ private boolean isTagLengthWithinLimits(String value) {
int maxLength = definition.getMaxTagLength();
return maxLength < 0 || value.length() <= maxLength;
}
+ private boolean isSimilarityTagWithinLimits(String value, int similarityTagCount) {
+ if (!isTagLengthWithinLimits(value)) {
+ return false;
+ }
+ int maxCount = definition.getMaxSimilarityTagsCount();
+ return maxCount < 0 || similarityTagCount < maxCount;
+ }
+
+ /**
+ * Process collected dynamic boost tags: sort by confidence (descending) and index only the top N
+ */
+ private boolean indexTopDynamicBoost(D doc, List<DynamicBoost> collectedBoosts, int maxCount) {
+ collectedBoosts.sort(Comparator.comparingDouble((DynamicBoost tag) -> tag.confidence).reversed());
+
+ int count = 0;
+ for (DynamicBoost tag : collectedBoosts) {
+ if (maxCount >= 0 && count >= maxCount) {
+ break;
+ }
+ try {
+ if (indexDynamicBoost(doc, tag.parent, tag.nodeName, tag.value, tag.confidence)) {
+ count++;
+ }
+ } catch (Exception e) {
+ log.error("Could not index dynamic boost tag '{}' for property {} and node {}", tag.value, tag.parent, tag.nodeName, e);
+ }
+ }
+
+ return count > 0;
+ }
+
/*
* Extracts the local name of the current node ignoring any namespace prefix
*/
@@ -771,4 +819,12 @@ public abstract class FulltextDocumentMaker<D> implements DocumentMaker<D> {
//cameCase file name to allow faster like search
indexNodeName(doc, value);
}
+
+ private static class DocumentBuildContext {
+ int similarityTagCount;
+ final List<DynamicBoost> collectedBoosts = new ArrayList<>();
+ }
+
+ private record DynamicBoost(String parent, String nodeName, String value, double confidence) {}
+
}
diff --git a/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/DynamicBoostCommonTest.java b/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/DynamicBoostCommonTest.java
index 6c5e6998a1..e14bace8a1 100644
--- a/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/DynamicBoostCommonTest.java
+++ b/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/DynamicBoostCommonTest.java
@@ -234,10 +234,11 @@ public abstract class DynamicBoostCommonTest extends AbstractQueryTest {
}
protected void createAssetsIndexAndProperties(boolean lite, boolean similarityTags, boolean useInFullTextQuery) throws Exception {
- createAssetsIndexAndProperties(lite, similarityTags, useInFullTextQuery, null);
+ createAssetsIndexAndProperties(lite, similarityTags, useInFullTextQuery, null, null);
}
- protected void createAssetsIndexAndProperties(boolean lite, boolean similarityTags, boolean useInFullTextQuery, Integer maxTagLength) throws Exception {
+ protected void createAssetsIndexAndProperties(boolean lite, boolean similarityTags, boolean useInFullTextQuery,
+ Integer maxTagLength, Integer maxTagCount) throws Exception {
NodeTypeRegistry.register(root, new ByteArrayInputStream(ASSET_NODE_TYPE.getBytes()), "test nodeType");
Tree indexRuleProps = createIndex("dam:Asset", lite);
@@ -259,6 +260,11 @@ public abstract class DynamicBoostCommonTest extends AbstractQueryTest {
indexDef.setProperty(FulltextIndexConstants.MAX_TAG_LENGTH, maxTagLength);
}
+ if (maxTagCount != null) {
+ Tree indexDef = root.getTree("/oak:index/" + TEST_INDEX_NAME);
+ indexDef.setProperty(FulltextIndexConstants.MAX_DYNAMIC_BOOST_COUNT, maxTagCount);
+ }
+
root.commit();
}
diff --git a/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinitionTest.java b/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinitionTest.java
index d74e82202b..c80f06ef0f 100644
--- a/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinitionTest.java
+++ b/oak-search/src/test/java/org/apache/jackrabbit/oak/plugins/index/search/IndexDefinitionTest.java
@@ -526,6 +526,32 @@ public class IndexDefinitionTest {
assertEquals(50, defn.getMaxTagLength());
}
+ @Test
+ public void maxSimilarityTagsCount() {
+ NodeBuilder defnb = newFTIndexDefinition(builder.child(INDEX_DEFINITIONS_NAME), "foo",
+ "lucene", Set.of(TYPENAME_STRING));
+ IndexDefinition defn = new IndexDefinition(root, defnb.getNodeState(), "/foo");
+ assertEquals(IndexDefinition.DEFAULT_MAX_SIMILARITY_TAGS_COUNT, defn.getMaxSimilarityTagsCount());
+
+ defnb.setProperty(FulltextIndexConstants.MAX_SIMILARITY_TAGS_COUNT, 100);
+
+ defn = new IndexDefinition(root, defnb.getNodeState(), "/foo");
+ assertEquals(100, defn.getMaxSimilarityTagsCount());
+ }
+
+ @Test
+ public void maxDynamicBoostCount() {
+ NodeBuilder defnb = newFTIndexDefinition(builder.child(INDEX_DEFINITIONS_NAME), "foo",
+ "lucene", Set.of(TYPENAME_STRING));
+ IndexDefinition defn = new IndexDefinition(root, defnb.getNodeState(), "/foo");
+ assertEquals(IndexDefinition.DEFAULT_MAX_DYNAMIC_BOOST_COUNT, defn.getMaxDynamicBoostCount());
+
+ defnb.setProperty(FulltextIndexConstants.MAX_DYNAMIC_BOOST_COUNT, 100);
+
+ defn = new IndexDefinition(root, defnb.getNodeState(), "/foo");
+ assertEquals(100, defn.getMaxDynamicBoostCount());
+ }
+
@Test
public void maxExtractLength() {
NodeBuilder defnb = newFTIndexDefinition(builder.child(INDEX_DEFINITIONS_NAME), "foo",