You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@solr.apache.org by mk...@apache.org on 2026/09/11 19:36:55 UTC
(solr) branch main updated: aux index join qparser - a prunable segment-parallel auxiliary index join (#4749)
This is an automated email from the ASF dual-hosted git repository.
mkhludnev pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/main by this push:
new a6d33d6ed41 aux index join qparser - a prunable segment-parallel auxiliary index join (#4749)
a6d33d6ed41 is described below
commit a6d33d6ed419a4c7c2c72cbca9b11319fa0332b9
Author: Mikhail Khludnev <mk...@apache.org>
AuthorDate: Fri Sep 11 22:36:49 2026 +0300
aux index join qparser - a prunable segment-parallel auxiliary index join (#4749)
Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
Co-authored-by: David Smiley <ds...@apache.org>
---
changelog/unreleased/auxindexjoin-qparser.yml | 10 +
.../search/join/AuxIndexJoinQParserPlugin.java | 451 +++++++++
.../join/auxindexjoin/AuxIndexJoinConfig.java | 207 ++++
.../join/auxindexjoin/AuxIndexJoinMergePolicy.java | 839 ++++++++++++++++
.../join/auxindexjoin/AuxIndexJoinQuery.java | 221 ++++
.../search/join/auxindexjoin/AuxIndexManager.java | 381 +++++++
.../join/auxindexjoin/DirectExecutorService.java | 64 ++
.../search/join/auxindexjoin/DocAlignedMerge.java | 331 ++++++
.../search/join/auxindexjoin/ForeignKeyColumn.java | 223 ++++
.../join/auxindexjoin/FromLeafJoinContext.java | 71 ++
.../search/join/auxindexjoin/JoinColumWriter.java | 42 +
.../join/auxindexjoin/JoinColumnDocWriter.java | 207 ++++
.../join/auxindexjoin/JoinColumnIndexer.java | 411 ++++++++
.../join/auxindexjoin/JoinIndexScorerSupplier.java | 941 +++++++++++++++++
.../search/join/auxindexjoin/JoinIndexUtils.java | 1061 ++++++++++++++++++++
.../search/join/auxindexjoin/JoinIndexWeight.java | 130 +++
.../search/join/auxindexjoin/package-info.java | 166 +++
.../solr/configsets/auxindexjoin/conf/schema.xml | 31 +
.../configsets/auxindexjoin/conf/solrconfig.xml | 30 +
.../DistribAuxIndexJoinFromCollectionTest.java | 230 +++++
.../search/join/TestAuxIndexJoinQParserPlugin.java | 232 +++++
.../join/auxindexjoin/AuxIndexJoinBenchmark.java | 377 +++++++
.../join/auxindexjoin/ParallelIndexSearcher.java | 34 +
.../search/join/auxindexjoin/TestAuxIndexJoin.java | 604 +++++++++++
.../auxindexjoin/TestAuxIndexJoinMergePolicy.java | 990 ++++++++++++++++++
.../join/auxindexjoin/TestJoinColumnModel.java | 226 +++++
.../pages/auxindexjoin-query-parser.adoc | 185 ++++
.../query-guide/pages/common-query-parameters.adoc | 4 +-
.../modules/query-guide/pages/other-parsers.adoc | 7 +
29 files changed, 8705 insertions(+), 1 deletion(-)
diff --git a/changelog/unreleased/auxindexjoin-qparser.yml b/changelog/unreleased/auxindexjoin-qparser.yml
new file mode 100644
index 00000000000..3548e368da7
--- /dev/null
+++ b/changelog/unreleased/auxindexjoin-qparser.yml
@@ -0,0 +1,10 @@
+title: >
+ Introducing {!auxIndexJoin} query for query-time join with auxiliary index.
+type: added
+authors:
+ - name: Mikhail Khludnev
+ nick: mkhl
+links:
+ - name: SOLR-18307
+ url: https://issues.apache.org/jira/browse/SOLR-18307
+
diff --git a/solr/core/src/java/org/apache/solr/search/join/AuxIndexJoinQParserPlugin.java b/solr/core/src/java/org/apache/solr/search/join/AuxIndexJoinQParserPlugin.java
new file mode 100644
index 00000000000..2d3f7b5cf68
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/AuxIndexJoinQParserPlugin.java
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.store.Directory;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.CloseHook;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.core.DirectoryFactory;
+import org.apache.solr.core.DirectoryFactory.DirContext;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.request.SolrRequestInfo;
+import org.apache.solr.response.QueryResponseWriter;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.QParser;
+import org.apache.solr.search.QParserPlugin;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SyntaxError;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexJoinConfig;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager;
+import org.apache.solr.util.RefCounted;
+import org.apache.solr.util.plugin.SolrCoreAware;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Query parser exercising {@link AuxIndexManager} inside a {@link SolrCore}: it mimics {@link
+ * ScoreJoinQParserPlugin}'s local parameters, but resolves matches through the sidecar join index
+ * instead of {@link org.apache.lucene.search.join.JoinUtil}. Local parameters:
+ *
+ * <ul>
+ * <li>from - "foreign key" field name, collected while enumerating the subordinate query (the
+ * local parameter value).
+ * <li>fromIndex - optional core name to run the subordinate query against, when it differs from
+ * this core; cross-core joins are the reason {@link AuxIndexManager} exists in the first
+ * place, so this mirrors {@link ScoreJoinQParserPlugin}'s <code>fromIndex</code>, including
+ * SolrCloud alias/collection resolution via {@link ScoreJoinQParserPlugin#getCoreName}.
+ * <li>to - "primary key" field name looked up in this core's index.
+ * </ul>
+ *
+ * Example: {@code q={!aijoin from=manu_id_s to=id fromIndex=products}foo}.
+ *
+ * <p>Unlike {@link ScoreJoinQParserPlugin.OtherCoreJoinQuery}, which only borrows the from-side
+ * searcher long enough to build a self-contained {@code Query} in {@code createWeight}, an {@link
+ * org.apache.solr.search.join.auxindexjoin.AuxIndexJoinQuery} keeps reading the from-side searcher
+ * on every {@code scorerSupplier} call (it may lazily build missing pair columns per to-segment),
+ * so a cross-core from-searcher is pinned open for the whole request via {@link
+ * SolrRequestInfo#addCloseHook}, the same mechanism {@link
+ * org.apache.solr.search.JoinQuery.JoinQueryWeight} uses for the regular {@code {!join}}.
+ *
+ * <p>One {@link AuxIndexManager} is opened per sidecar directory in {@link #inform(SolrCore)},
+ * backed by a directory under the core's dataDir (configurable via the {@code dir} init parameter,
+ * resolved relative to dataDir unless absolute), and closed when the last core using it closes --
+ * see {@link SharedManagers} for why it outlives a single core. The remaining init parameters
+ * mirror {@link AuxIndexJoinConfig}: {@code singleFieldPerSegment}, {@code blockingRefresh}, {@code
+ * useFromSideThreads}, {@code wipeOnVersionMismatch}, {@code sweepSamplingInterval} (seconds), and
+ * the sidecar's compaction knobs {@code mergeSegmentsAtOnce}, {@code maxPairsPerSegment} and {@code
+ * minDeadPercentToPurge}. This sidecar always belongs to the "to" side core -- the one this plugin
+ * is registered in.
+ *
+ * <p><b>Why this implements {@link QueryResponseWriter}:</b> {@link
+ * org.apache.solr.core.SolrResourceLoader}'s {@code awareCompatibility} allowlist (see SOLR-8311)
+ * only lets specific plugin base types implement {@link SolrCoreAware}, and {@code QParserPlugin}
+ * isn't one of them, so a plain {@code implements SolrCoreAware} fails core load with "Invalid
+ * 'Aware' object". {@code QueryResponseWriter} is on the allowlist and happens to be the cheapest
+ * interface there to satisfy (two abstract methods, both unreachable stubs below -- this class is
+ * never registered as a {@code <queryResponseWriter>}). This is safe here specifically because
+ * {@code QParserPlugin} instances are loaded once per core load/reload via {@link
+ * org.apache.solr.core.PluginBag}, exactly like the already-whitelisted {@link
+ * org.apache.solr.handler.component.SearchComponent} -- never created ad-hoc per request ({@link
+ * QParser#getParser(String, SolrQueryRequest)} resolves the already registered instance via {@code
+ * req.getCore().getQueryPlugin(name)}).
+ *
+ * @lucene.experimental
+ * <p>This is experimental API and subject to change.
+ */
+public class AuxIndexJoinQParserPlugin extends QParserPlugin
+ implements QueryResponseWriter, SolrCoreAware {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Init parameter: directory holding the sidecar join index, resolved against the core's dataDir
+ * unless absolute. Defaults to {@value #DEFAULT_DIR}.
+ */
+ public static final String DIR = "dir";
+
+ public static final String NAME = "auxIndexJoin";
+
+ public static final String DEFAULT_DIR = "aux-index-join";
+
+ /** Init parameter: whether each pair column is flushed into its own sidecar segment. */
+ public static final String SINGLE_FIELD_PER_SEGMENT = "singleFieldPerSegment";
+
+ /** Init parameter: whether writing a batch of pair columns blocks until it is searchable. */
+ public static final String BLOCKING_REFRESH = "blockingRefresh";
+
+ /**
+ * Init parameter: whether loading from-side leaves that feed a join build is parallelized across
+ * executor threads. Defaults to {@code true}.
+ */
+ public static final String USE_FROM_SIDE_THREADS = "useFromSideThreads";
+
+ /**
+ * Init parameter: whether opening the join index wipes a pre-existing on-disk index recorded with
+ * a different Solr major version. Defaults to {@code true}.
+ */
+ public static final String WIPE_ON_VERSION_MISMATCH = "wipeOnVersionMismatch";
+
+ /**
+ * Init parameter: how often (in seconds) {@code AuxIndexManager.onCreateWeight} samples searcher
+ * state for the dead-pair reaper. Non-positive means sample on every call. Defaults to 60.
+ */
+ public static final String SWEEP_SAMPLING_INTERVAL = "sweepSamplingInterval";
+
+ /**
+ * Init parameter: how many sidecar segments one compaction folds into a single doc-aligned one.
+ * Also the floor below which no compaction is proposed at all, so it belongs below the segment
+ * count the sidecar runs at. See {@link AuxIndexJoinConfig#setMergeSegmentsAtOnce}.
+ */
+ public static final String MERGE_SEGMENTS_AT_ONCE = "mergeSegmentsAtOnce";
+
+ /**
+ * Init parameter: how many pair columns a sidecar segment may already carry and still be
+ * compacted again -- what makes compaction converge. See {@link
+ * AuxIndexJoinConfig#setMaxPairsPerSegment}.
+ */
+ public static final String MAX_PAIRS_PER_SEGMENT = "maxPairsPerSegment";
+
+ /**
+ * Init parameter: how many bytes a purge has to reclaim before it rewrites a segment just to drop
+ * the columns dead in it. See {@link AuxIndexJoinConfig#setMinReclaimableBytesToPurge}.
+ */
+ public static final String MIN_RECLAIMABLE_BYTES_TO_PURGE = "minReclaimableBytesToPurge";
+
+ /**
+ * Init parameter: how often, in milliseconds, the sidecar is committed -- writing a batch only
+ * flushes it. See {@link AuxIndexJoinConfig#setCommitIntervalMs}.
+ */
+ public static final String COMMIT_INTERVAL_MS = "commitIntervalMs";
+
+ private String configuredDir = DEFAULT_DIR;
+ private final AuxIndexJoinConfig joinIndexConfig = new AuxIndexJoinConfig();
+
+ private volatile AuxIndexManager joinIndex;
+
+ /** The sidecar this core's joins run through; package-private for tests. */
+ AuxIndexManager getJoinIndex() {
+ return joinIndex;
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ if (args != null) {
+ SolrParams params = args.toSolrParams();
+ configuredDir = params.get(DIR, DEFAULT_DIR);
+ joinIndexConfig.setSingleFieldPerSegment(
+ params.getBool(SINGLE_FIELD_PER_SEGMENT, joinIndexConfig.getSingleFieldPerSegment()));
+ joinIndexConfig.setBlockingRefresh(
+ params.getBool(BLOCKING_REFRESH, joinIndexConfig.getBlockingRefresh()));
+ joinIndexConfig.setUseFromSideThreads(
+ params.getBool(USE_FROM_SIDE_THREADS, joinIndexConfig.getUseFromSideThreads()));
+ joinIndexConfig.setWipeOnVersionMismatch(
+ params.getBool(WIPE_ON_VERSION_MISMATCH, joinIndexConfig.getWipeOnVersionMismatch()));
+ joinIndexConfig.setSweepSamplingInterval(
+ params.getLong(SWEEP_SAMPLING_INTERVAL, 60), TimeUnit.SECONDS);
+ joinIndexConfig.setMergeSegmentsAtOnce(
+ params.getInt(MERGE_SEGMENTS_AT_ONCE, joinIndexConfig.getMergeSegmentsAtOnce()));
+ joinIndexConfig.setMaxPairsPerSegment(
+ params.getInt(MAX_PAIRS_PER_SEGMENT, joinIndexConfig.getMaxPairsPerSegment()));
+ joinIndexConfig.setMinReclaimableBytesToPurge(
+ params.getLong(
+ MIN_RECLAIMABLE_BYTES_TO_PURGE, joinIndexConfig.getMinReclaimableBytesToPurge()));
+ joinIndexConfig.setCommitIntervalMs(
+ params.getLong(COMMIT_INTERVAL_MS, joinIndexConfig.getCommitIntervalMs()));
+ }
+ }
+
+ @Override
+ public void inform(SolrCore core) {
+ Path path = Path.of(configuredDir);
+ if (!path.isAbsolute()) {
+ path = Path.of(core.getDataDir()).resolve(path);
+ } else {
+ core.getCoreContainer().assertPathAllowed(path);
+ }
+ final String sidecarPath = path.toAbsolutePath().normalize().toString();
+ final SharedManagers sidecars = SharedManagers.of(core);
+ try {
+ joinIndex = sidecars.acquire(sidecarPath, core, joinIndexConfig);
+ } catch (IOException | RuntimeException e) {
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Failed to open AuxIndexManager at " + path, e);
+ }
+ core.addCloseHook(
+ new CloseHook() {
+ @Override
+ public void preClose(SolrCore core) {
+ sidecars.release(sidecarPath);
+ }
+ });
+ }
+
+ /**
+ * The {@link AuxIndexManager}s open in one {@link CoreContainer}, keyed by sidecar directory and
+ * ref-counted by the cores using them.
+ *
+ * <p>Sharing is what makes a core reload work at all. The sidecar's {@link
+ * org.apache.lucene.index.IndexWriter} holds a {@code write.lock} that is exclusive per path
+ * within a JVM, and a reload fully builds the new {@link SolrCore} -- running {@link
+ * #inform(SolrCore)} on this plugin's new instance -- before closing the old one, so for a while
+ * both cores want the same sidecar. Opening a second manager there fails with {@link
+ * org.apache.lucene.store.LockObtainFailedException}, and with it the reload; joining the open
+ * one instead lets the last core to close it release the writer and the directory. That mirrors
+ * how a reload passes the main index' {@link org.apache.solr.update.SolrCoreState}, and with it
+ * the live {@code IndexWriter}, on to the new core; it also keeps the pair columns built so far
+ * warm across the reload, which is desirable anyway since a reload changes no doc ids.
+ *
+ * <p>This registry has to outlive a single {@link SolrCore}, but nothing longer, so it lives in
+ * the {@link CoreContainer}'s {@link org.apache.solr.common.util.ObjectCache}: the container is
+ * the same object across a reload, and {@code CoreContainer#shutdown} closes every core before
+ * closing that cache, so the close hooks above always run first and this {@link #close()} only
+ * ever sees leaks. Compare {@code CachingDirectoryFactory#byPathCache}, which ref-counts
+ * Directories by path the same way and survives a reload by living on the {@code
+ * DirectoryFactory} that the new core inherits from the shared {@code SolrCoreState}.
+ *
+ * <p>The {@link AuxIndexJoinConfig} of the core that opened a manager stays in effect: a reload
+ * changing the init parameters only takes them into account once the sidecar is closed and
+ * reopened, i.e. when every core using it has been unloaded.
+ */
+ static final class SharedManagers implements Closeable {
+
+ private static final String OBJECT_CACHE_KEY =
+ AuxIndexJoinQParserPlugin.class.getName() + ".sidecars";
+
+ private final Map<String, SharedManager> open = new HashMap<>();
+
+ /** The registry of the core's container, created on first use. */
+ static SharedManagers of(SolrCore core) {
+ return core.getCoreContainer()
+ .getObjectCache()
+ .computeIfAbsent(OBJECT_CACHE_KEY, SharedManagers.class, key -> new SharedManagers());
+ }
+
+ /** Opens the sidecar at the given path, or joins the manager already open there. */
+ synchronized AuxIndexManager acquire(String path, SolrCore core, AuxIndexJoinConfig config)
+ throws IOException {
+ SharedManager shared = open.get(path);
+ if (shared != null) {
+ shared.refCount++;
+ if (!Objects.equals(shared.directoryFactory, core.getDirectoryFactory())) {
+ // a reload that doesn't pass its SolrCoreState on (the index dir changed) brings a new
+ // DirectoryFactory: the sidecar keeps the directory it was opened with, which the old
+ // factory closes when the old core goes away
+ log.warn(
+ "AuxIndexManager at {} is shared with a core using another DirectoryFactory; "
+ + "reload this core once the previous one is closed if the sidecar stops working",
+ path);
+ }
+ log.info(
+ "Joining the AuxIndexManager open at {}, now used by {} cores", path, shared.refCount);
+ return shared.manager;
+ }
+ DirectoryFactory directoryFactory = core.getDirectoryFactory();
+ Directory directory =
+ directoryFactory.get(path, DirContext.DEFAULT, core.getSolrConfig().indexConfig.lockType);
+ try {
+ AuxIndexManager manager = new AuxIndexManager(directory, config);
+ open.put(path, new SharedManager(manager, directoryFactory, directory));
+ return manager;
+ } catch (IOException | RuntimeException e) {
+ try {
+ directoryFactory.release(directory);
+ } catch (IOException releaseException) {
+ e.addSuppressed(releaseException);
+ }
+ throw e;
+ }
+ }
+
+ /** Drops one core's usage of the sidecar at the given path, closing it when it was the last. */
+ synchronized void release(String path) {
+ SharedManager shared = open.get(path);
+ if (shared == null) {
+ log.warn("No AuxIndexManager open at {} to release", path);
+ return;
+ }
+ if (--shared.refCount > 0) {
+ log.info("Keeping the AuxIndexManager at {} open for {} cores", path, shared.refCount);
+ return;
+ }
+ open.remove(path);
+ shared.close();
+ }
+
+ /**
+ * Closes whatever is left when the container's ObjectCache goes down. Every core has been
+ * closed by then, so an entry surviving to here means a close hook didn't run -- close it
+ * rather than leak the writer past the container.
+ */
+ @Override
+ public synchronized void close() {
+ for (Map.Entry<String, SharedManager> entry : open.entrySet()) {
+ log.warn("AuxIndexManager at {} was still open at container shutdown", entry.getKey());
+ entry.getValue().close();
+ }
+ open.clear();
+ }
+
+ /** One open sidecar: the manager, what it was opened with, and the cores using it. */
+ private static final class SharedManager {
+
+ private final AuxIndexManager manager;
+ private final DirectoryFactory directoryFactory;
+ private final Directory directory;
+ private int refCount = 1;
+
+ private SharedManager(
+ AuxIndexManager manager, DirectoryFactory directoryFactory, Directory directory) {
+ this.manager = manager;
+ this.directoryFactory = directoryFactory;
+ this.directory = directory;
+ }
+
+ private void close() {
+ try {
+ manager.close();
+ } catch (IOException e) {
+ log.warn("Failed closing AuxIndexManager", e);
+ } finally {
+ try {
+ directoryFactory.release(directory);
+ } catch (IOException e) {
+ log.warn("Failed releasing AuxIndexManager directory {}", directory, e);
+ }
+ }
+ }
+ }
+ }
+
+ // QueryResponseWriter stubs, unreachable: implemented only to satisfy SolrCoreAware's allowlist,
+ // see the class javadoc. This plugin is never registered as a <queryResponseWriter>.
+
+ @Override
+ public void write(
+ OutputStream out, SolrQueryRequest request, SolrQueryResponse response, String contentType) {
+ throw new UnsupportedOperationException(
+ AuxIndexJoinQParserPlugin.class.getSimpleName()
+ + " is a QParserPlugin, not a QueryResponseWriter");
+ }
+
+ @Override
+ public String getContentType(SolrQueryRequest request, SolrQueryResponse response) {
+ throw new UnsupportedOperationException(
+ AuxIndexJoinQParserPlugin.class.getSimpleName()
+ + " is a QParserPlugin, not a QueryResponseWriter");
+ }
+
+ @Override
+ public QParser createParser(
+ String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
+ return new QParser(qstr, localParams, params, req) {
+ @Override
+ public Query parse() throws SyntaxError {
+ if (joinIndex == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "AuxIndexJoinQParserPlugin is not initialized; is it registered as a <queryParser>?");
+ }
+ final String fromField = getParam("from");
+ final String toField = getParam("to");
+ if (fromField == null || toField == null) {
+ throw new SyntaxError("auxIndexJoin query parser requires 'from' and 'to' local params");
+ }
+ final String fromIndex = localParams.get("fromIndex");
+ final String v = localParams.get(CommonParams.VALUE);
+ final String myCore = req.getCore().getCoreDescriptor().getName();
+
+ final Query fromQuery;
+ final IndexSearcher fromSearcher;
+ ExecutorService fromExecutor;
+ if (fromIndex != null && !fromIndex.equals(myCore)) {
+ CoreContainer container = req.getCoreContainer();
+ String coreName =
+ ScoreJoinQParserPlugin.getCoreName(
+ fromIndex, container, req.getCore(), toField, fromField, localParams);
+ SolrCore fromCore = container.getCore(coreName);
+ if (fromCore == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Cross-core join: no such core " + coreName);
+ }
+ SolrRequestInfo info = SolrRequestInfo.getRequestInfo();
+ if (info == null) {
+ fromCore.close();
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Cross-core auxIndexJoin must have SolrRequestInfo");
+ }
+ // released once this request completes: the from-side searcher is read on every
+ // scorerSupplier() call, not just while building this query, so it must outlive parse()
+ info.addCloseHook(fromCore);
+ try (SolrQueryRequestBase otherReq = new SolrQueryRequestBase(fromCore, params)) {
+ fromQuery = QParser.getParser(v, otherReq).getQuery();
+ }
+ RefCounted<SolrIndexSearcher> fromRef = fromCore.getSearcher(false, true, null);
+ info.addCloseHook(fromRef::decref);
+ fromSearcher = fromRef.get();
+ fromExecutor = (ExecutorService) fromCore.getCoreContainer().getIndexSearcherExecutor();
+ } else {
+ fromQuery = subQuery(v, null).getQuery();
+ fromSearcher = req.getSearcher();
+ fromExecutor = (ExecutorService) req.getCoreContainer().getIndexSearcherExecutor();
+ }
+
+ return joinIndex.newJoinQuery(fromField, fromQuery, fromSearcher, toField, fromExecutor);
+ }
+ };
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinConfig.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinConfig.java
new file mode 100644
index 00000000000..a998337be49
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinConfig.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.lucene.store.Directory;
+
+/**
+ * Holds the configuration used to create an {@link AuxIndexManager}. Every setter returns {@link
+ * AuxIndexJoinConfig} to allow chaining settings conveniently, for example:
+ *
+ * <pre class="prettyprint">
+ * AuxIndexJoinConfig config = new AuxIndexJoinConfig().setBlockingRefresh(false);
+ * AuxIndexManager joinIndex = new AuxIndexManager(joinDir, config);
+ * </pre>
+ *
+ * <p>Once passed to {@link AuxIndexManager#AuxIndexManager(Directory, AuxIndexJoinConfig)}, changes
+ * to this object no longer affect the created {@link AuxIndexManager} instance.
+ *
+ * @lucene.experimental
+ * <p>This is experimental API and subject to change.
+ */
+public final class AuxIndexJoinConfig {
+
+ private boolean singleFieldPerSegment = false;
+ private boolean blockingRefresh = true;
+ private boolean useFromSideThreads = true;
+ private boolean wipeOnVersionMismatch = true;
+ private long sweepSamplingIntervalNanos = TimeUnit.MINUTES.toNanos(1);
+ private int mergeSegmentsAtOnce = AuxIndexJoinMergePolicy.DEFAULT_MERGE_SEGMENTS_AT_ONCE;
+ private int maxPairsPerSegment = AuxIndexJoinMergePolicy.DEFAULT_MAX_PAIRS_PER_SEGMENT;
+ private long minReclaimableBytesToPurge =
+ AuxIndexJoinMergePolicy.DEFAULT_MIN_RECLAIMABLE_BYTES_TO_PURGE;
+ private long commitIntervalMs = TimeUnit.SECONDS.toMillis(30);
+
+ /** Sole constructor, using the default settings documented on each setter. */
+ public AuxIndexJoinConfig() {}
+
+ /**
+ * Whether each pair column is flushed into its own sidecar segment, rather than batching every
+ * pair column built in the same round into one segment. Default is {@code false}: many columns
+ * per segment, traded off against a longer sweep to reclaim any that become dead.
+ */
+ public AuxIndexJoinConfig setSingleFieldPerSegment(boolean singleFieldPerSegment) {
+ this.singleFieldPerSegment = singleFieldPerSegment;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setSingleFieldPerSegment}. */
+ public boolean getSingleFieldPerSegment() {
+ return singleFieldPerSegment;
+ }
+
+ /**
+ * Whether writing a batch of pair columns blocks until the sidecar's {@link
+ * org.apache.lucene.search.SearcherManager} is refreshed past it, so the freshly built pairs are
+ * visible to the caller that triggered the build. Default is {@code true}.
+ */
+ public AuxIndexJoinConfig setBlockingRefresh(boolean blockingRefresh) {
+ this.blockingRefresh = blockingRefresh;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setBlockingRefresh}. */
+ public boolean getBlockingRefresh() {
+ return blockingRefresh;
+ }
+
+ /**
+ * Whether loading the from-side leaves that feed a join build is parallelized across the caller's
+ * executor threads. When {@code true} (the default) each from-side segment is loaded on a
+ * separate executor thread; when {@code false} they are loaded sequentially on the calling
+ * thread. Set to {@code false} to bound the load to a single thread, e.g. when the executor is
+ * contended or to keep queries deterministic.
+ */
+ public AuxIndexJoinConfig setUseFromSideThreads(boolean useFromSideThreads) {
+ this.useFromSideThreads = useFromSideThreads;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setUseFromSideThreads}. */
+ public boolean getUseFromSideThreads() {
+ return useFromSideThreads;
+ }
+
+ /**
+ * Whether opening the join index wipes a pre-existing on-disk index whose commit user data
+ * records a different Solr major version than the current one (see {@code
+ * AuxIndexManager#wipeIfIncompatibleVersion}). Default is {@code true}: doc-id mappings are
+ * version-specific and stale pairs are rebuilt lazily, so keeping them risks reading corrupt
+ * joins after an upgrade. Disable only when the same process is guaranteed to own the index.
+ */
+ public AuxIndexJoinConfig setWipeOnVersionMismatch(boolean wipeOnVersionMismatch) {
+ this.wipeOnVersionMismatch = wipeOnVersionMismatch;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setWipeOnVersionMismatch}. */
+ public boolean getWipeOnVersionMismatch() {
+ return wipeOnVersionMismatch;
+ }
+
+ /**
+ * How often {@link AuxIndexManager#onCreateWeight} actually samples searcher state for the
+ * dead-pair reaper; calls arriving sooner than this after the last accepted sample are skipped,
+ * since sampling is only a heuristic hint feeding the reap decision, not a correctness
+ * requirement. Default is one minute. Pass zero (or a non-positive value) to sample on every
+ * call.
+ */
+ public AuxIndexJoinConfig setSweepSamplingInterval(long duration, TimeUnit unit) {
+ this.sweepSamplingIntervalNanos = unit.toNanos(duration);
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setSweepSamplingInterval}, in nanoseconds. */
+ public long getSweepSamplingIntervalNanos() {
+ return sweepSamplingIntervalNanos;
+ }
+
+ /**
+ * How many sidecar segments one compaction folds into a single doc-aligned one. This is a floor
+ * as much as a batch size: with fewer eligible segments than this, {@code
+ * AuxIndexJoinMergePolicy} proposes no compaction at all, so it has to sit below the segment
+ * count the sidecar actually runs at. Default is 4.
+ */
+ public AuxIndexJoinConfig setMergeSegmentsAtOnce(int mergeSegmentsAtOnce) {
+ this.mergeSegmentsAtOnce = mergeSegmentsAtOnce;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setMergeSegmentsAtOnce}. */
+ public int getMergeSegmentsAtOnce() {
+ return mergeSegmentsAtOnce;
+ }
+
+ /**
+ * How many pair columns a sidecar segment may already carry and still be compacted again. Merged
+ * segments grow wider, not longer, so this is what makes compaction converge instead of rewriting
+ * the same big segment forever. Default is 1024.
+ */
+ public AuxIndexJoinConfig setMaxPairsPerSegment(int maxPairsPerSegment) {
+ this.maxPairsPerSegment = maxPairsPerSegment;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setMaxPairsPerSegment}. */
+ public int getMaxPairsPerSegment() {
+ return maxPairsPerSegment;
+ }
+
+ /**
+ * How many bytes a purge has to reclaim before it rewrites a segment just to drop the columns
+ * that are dead in it. Ranking by bytes rather than by share of columns is what keeps the widest
+ * segments -- which dilute their own dead share, and hold most of the sidecar -- from being the
+ * least likely to ever be cleaned. Default is 1MB. Zero purges on any death at all; compaction is
+ * not subject to it, having decided to rewrite anyway.
+ */
+ public AuxIndexJoinConfig setMinReclaimableBytesToPurge(long minReclaimableBytesToPurge) {
+ this.minReclaimableBytesToPurge = minReclaimableBytesToPurge;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setMinReclaimableBytesToPurge}. */
+ public long getMinReclaimableBytesToPurge() {
+ return minReclaimableBytesToPurge;
+ }
+
+ /**
+ * How often the sidecar is committed, in milliseconds. Writing a batch only flushes it -- that
+ * seals the batch into a segment of its own, which is what a pair column needs to start at doc 0
+ * -- so this is what fsyncs those segments and moves the commit point forward.
+ *
+ * <p>Durability is barely the point: the sidecar is derived, and pairs lost to an unclean stop
+ * are rebuilt on the next query exactly as pairs that were never built are. Reclaiming files is:
+ * Lucene retains every file the last commit point references, so this interval sets how far above
+ * its live size the sidecar directory floats.
+ *
+ * <p>Default is 30 seconds. Zero commits at the end of every batch instead -- durable per batch,
+ * at the cost of an fsync on the query thread, holding the write lock, once per batch.
+ */
+ public AuxIndexJoinConfig setCommitIntervalMs(long commitIntervalMs) {
+ if (commitIntervalMs < 0) {
+ throw new IllegalArgumentException("commitIntervalMs must not be negative");
+ }
+ this.commitIntervalMs = commitIntervalMs;
+ return this;
+ }
+
+ /** Returns the current value set via {@link #setCommitIntervalMs}. */
+ public long getCommitIntervalMs() {
+ return commitIntervalMs;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinMergePolicy.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinMergePolicy.java
new file mode 100644
index 00000000000..2d1014eba87
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinMergePolicy.java
@@ -0,0 +1,839 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.MergePolicy;
+import org.apache.lucene.index.MergeTrigger;
+import org.apache.lucene.index.SegmentCommitInfo;
+import org.apache.lucene.index.SegmentInfos;
+import org.apache.lucene.search.IndexSearcher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * @lucene.experimental
+ */
+final class AuxIndexJoinMergePolicy extends MergePolicy {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * How many sidecar segments one {@link DocAlignedMerge} folds into one, by default.
+ *
+ * <p>This is a floor as much as a batch size -- with fewer eligible segments than this, no
+ * compaction is proposed at all -- so it has to sit below the segment count the sidecar actually
+ * runs at, not above it. At 10 it never fired once in a 41-minute run: the sidecar sat at 7-16
+ * segments, of which at most 9 were ever eligible at the same time (the rest merging, deleted, or
+ * over {@link #DEFAULT_MAX_PAIRS_PER_SEGMENT}), so every round logged "nothing to do" while pairs
+ * queued for removal piled up until the heap gave out.
+ */
+ static final int DEFAULT_MERGE_SEGMENTS_AT_ONCE = 4;
+
+ /**
+ * How many pair columns a sidecar segment may already carry and still be compacted again, by
+ * default. Merged segments grow wider, not longer, so this is what makes compaction converge
+ * instead of rewriting the same big segment forever.
+ */
+ static final int DEFAULT_MAX_PAIRS_PER_SEGMENT = 1024;
+
+ /**
+ * How many bytes a purge has to reclaim before it is worth rewriting the segment for, by default.
+ *
+ * <p>A purge pays the segment's whole size to drop what is dead in it, so it has to earn the
+ * trip; the question is what "earning it" should measure. A share of the columns -- one dead in
+ * ten, say -- reads well and is exactly wrong for this index: dead columns land in whatever
+ * segment compaction last folded them into, so the widest segments dilute their own dead share
+ * below any fixed percentage and become the least likely to ever be cleaned, though they are
+ * where the bytes are. A 41-minute run ended with a 96MB sidecar of which 74MB was one segment
+ * carrying, at roughly 8% dead, some 6MB nobody could read: under the share rule every round
+ * logged "too little dead to be worth rewriting", and the queue never drained below 197 names.
+ *
+ * <p>Bytes rank those the way an operator would. At 1MB the 74MB segment above is rewritten to
+ * win 6MB back, while a 1.3MB segment at the same 8% offers 100KB and is left alone until more of
+ * it dies. The estimate assumes columns in a segment cost about the same, which is only roughly
+ * true -- it decides an order of magnitude, not a byte count.
+ */
+ static final long DEFAULT_MIN_RECLAIMABLE_BYTES_TO_PURGE = 1L << 20;
+
+ /**
+ * Cap on the merges proposed in one round. A backlog of hundreds of segments is worked off over
+ * the following rounds (every flush and every finished merge triggers one) rather than handed to
+ * the scheduler at once, where it would stall the writing thread -- which here is a query thread
+ * building a pair column.
+ */
+ private static final int MAX_MERGES_PER_ROUND = 4;
+
+ private volatile int mergeSegmentsAtOnce = DEFAULT_MERGE_SEGMENTS_AT_ONCE;
+ private volatile int maxPairsPerSegment = DEFAULT_MAX_PAIRS_PER_SEGMENT;
+ private volatile long minReclaimableBytesToPurge = DEFAULT_MIN_RECLAIMABLE_BYTES_TO_PURGE;
+
+ /**
+ * One segment worth purging: what is dead in it, and how wide it is -- the two numbers that
+ * decide whether rewriting it pays, and which of several candidates pays best.
+ */
+ private record PurgeCandidate(
+ SegmentCommitInfo info, Set<String> dead, int width, long sizeInBytes) {
+
+ /** The share of this segment's columns that a purge would reclaim. */
+ double deadShare() {
+ return width == 0 ? 0 : (double) dead.size() / width;
+ }
+
+ /** What that share is worth in bytes, columns taken to cost about the same as each other. */
+ long reclaimableBytes() {
+ return (long) (sizeInBytes * deadShare());
+ }
+
+ boolean earnsRewrite(long minReclaimableBytes) {
+ return reclaimableBytes() >= minReclaimableBytes;
+ }
+ }
+
+ /**
+ * Last round's pair field names by segment key, so a steady stream of commits doesn't re-read
+ * every segment's FieldInfos -- a compound-file open and mmap per segment per round -- when only
+ * the newest segment is new. Rebuilt on every round, which is also what evicts gone segments.
+ * Written only by {@link #findMerges}, which is always called under the {@link IndexWriter}
+ * monitor, as its contract states; volatile because {@link #onCreateWeight} reads it from query
+ * threads to know what columns the sidecar actually holds.
+ */
+ private volatile Map<String, Set<String>> pairFieldNamesBySegment = Map.of();
+
+ /**
+ * Compaction knobs, fed from {@link AuxIndexJoinConfig} the way {@link #setSweepInterval} is: see
+ * the DEFAULT_ constants above for what each one trades off.
+ */
+ void setCompaction(int mergeSegmentsAtOnce, int maxPairsPerSegment) {
+ this.mergeSegmentsAtOnce = mergeSegmentsAtOnce;
+ this.maxPairsPerSegment = maxPairsPerSegment;
+ }
+
+ /**
+ * How many bytes a purge must reclaim before it rewrites a segment for that alone; see {@link
+ * #DEFAULT_MIN_RECLAIMABLE_BYTES_TO_PURGE}. Zero purges on any death at all.
+ */
+ void setMinReclaimableBytesToPurge(long minReclaimableBytesToPurge) {
+ this.minReclaimableBytesToPurge = minReclaimableBytesToPurge;
+ }
+
+ @Override
+ public MergePolicy.MergeSpecification findMerges(
+ MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergeContext mergeContext)
+ throws IOException {
+ Set<SegmentCommitInfo> merging = mergeContext.getMergingSegments();
+ Map<String, Set<String>> pairNamesThisRound = HashMap.newHashMap(segmentInfos.asList().size());
+ MergeSpecification spec = null;
+ List<SegmentCommitInfo> compactable = new ArrayList<>();
+ // one snapshot for the whole round, so every merge it proposes reaps a consistent set and no
+ // merge drains a name whose column another merge was going to drop
+ Set<String> reapSnapshot = Set.copyOf(pendingPairRemovals);
+ Map<SegmentCommitInfo, PurgeCandidate> reapableBySegment = new LinkedHashMap<>();
+ int dead = 0;
+ for (SegmentCommitInfo info : segmentInfos) {
+ Set<String> pairFieldNames = pairFieldNames(info, pairNamesThisRound);
+ if (merging.contains(info)) {
+ continue;
+ }
+ if (!pairFieldNames.isEmpty() && reapSnapshot.containsAll(pairFieldNames)) {
+ dead++;
+ spec = added(spec, new DropSegmentMerge(List.of(info), pairFieldNames));
+ continue;
+ }
+ Set<String> reapable = intersection(pairFieldNames, reapSnapshot);
+ if (!reapable.isEmpty() && isRewritable(info)) {
+ reapableBySegment.put(
+ info, new PurgeCandidate(info, reapable, pairFieldNames.size(), info.sizeInBytes()));
+ }
+ if (isCompactable(info, pairFieldNames)) {
+ compactable.add(info);
+ }
+ }
+ this.pairFieldNamesBySegment = pairNamesThisRound;
+ int aligned = 0;
+ Set<SegmentCommitInfo> taken = new HashSet<>();
+ if (compactable.size() >= mergeSegmentsAtOnce) {
+ // by maxDoc, so a merge groups sidecar segments of comparable length: the union is as long
+ // as its longest input, and every shorter input pays a bit of sparse-docvalues overhead for
+ // the docs it doesn't reach
+ compactable.sort(Comparator.comparingInt(info -> info.info.maxDoc()));
+ for (int from = 0;
+ from + mergeSegmentsAtOnce <= compactable.size() && aligned < MAX_MERGES_PER_ROUND;
+ from += mergeSegmentsAtOnce) {
+ List<SegmentCommitInfo> group =
+ List.copyOf(compactable.subList(from, from + mergeSegmentsAtOnce));
+ // no threshold here, unlike the purges below: these segments are being rewritten anyway,
+ // so dropping whatever of them is dead costs nothing extra
+ Set<String> reaped = new HashSet<>();
+ for (SegmentCommitInfo info : group) {
+ PurgeCandidate candidate = reapableBySegment.get(info);
+ if (candidate != null) {
+ reaped.addAll(candidate.dead());
+ }
+ }
+ taken.addAll(group);
+ aligned++;
+ spec =
+ added(
+ spec,
+ new DocAlignedMerge(
+ group, reaped, alignedMergeCount::incrementAndGet, this::drainReaped));
+ }
+ }
+ // Reaping on its own account. Compaction alone cannot be relied on to carry it: it needs a
+ // quorum of eligible segments, and it passes over exactly the segments most worth purging --
+ // isCompactable excludes anything already wider than maxPairsPerSegment. A segment holding dead
+ // columns is worth rewriting for that reason alone, so purge it as a merge of one. This
+ // converges: the merge drains those names, so the next round finds nothing reapable there.
+ //
+ // A purge exists only to drop columns, so unlike a compaction it has to earn its rewrite: it
+ // costs the segment's whole size to reclaim the bytes that are dead in it. Below the threshold
+ // the names simply stay queued and the next death in that segment is what pays for the trip --
+ // unless the queue itself has grown dangerous, at which point I/O is the cheaper currency and
+ // the threshold is waived.
+ boolean queueUnderPressure = pendingPairRemovals.size() >= PENDING_PURGE_HIGH_WATER;
+ List<PurgeCandidate> candidates = new ArrayList<>(reapableBySegment.values());
+ // most bytes reclaimed first: the per-round merge budget is small, so spend it where it buys
+ // the most, not on whichever segment happens to come first in the index
+ candidates.sort(Comparator.comparingLong(PurgeCandidate::reclaimableBytes).reversed());
+ int purged = 0;
+ int notWorthIt = 0;
+ PurgeCandidate bestDeclined = null;
+ for (PurgeCandidate candidate : candidates) {
+ if (aligned + purged >= MAX_MERGES_PER_ROUND) {
+ break;
+ }
+ if (taken.contains(candidate.info())) {
+ continue; // a compaction above is already dropping these
+ }
+ if (!queueUnderPressure && !candidate.earnsRewrite(minReclaimableBytesToPurge)) {
+ notWorthIt++;
+ // candidates are sorted by what they reclaim, so the first one declined is the best of
+ // them: report it, or "too little dead" is a verdict with no evidence attached
+ if (bestDeclined == null) {
+ bestDeclined = candidate;
+ }
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=purgeDeclined segment={} dead={} of {} columns ({}%),"
+ + " reclaims={} of {} bytes, threshold={}",
+ candidate.info().info.name,
+ candidate.dead().size(),
+ candidate.width(),
+ Math.round(candidate.deadShare() * 100),
+ candidate.reclaimableBytes(),
+ candidate.sizeInBytes(),
+ minReclaimableBytesToPurge);
+ continue;
+ }
+ purged++;
+ spec =
+ added(
+ spec,
+ new DocAlignedMerge(
+ List.of(candidate.info()),
+ candidate.dead(),
+ purgedSegmentCount::incrementAndGet,
+ this::drainReaped));
+ }
+ if (spec != null) {
+ if (log.isInfoEnabled()) {
+ log.info(
+ "AUXIJOIN sidecar compaction: {} doc-aligned merge(s) of {} segments each, {} column "
+ + "purge(s) and {} dead segment(s) dropped, out of {} segments ({} compactable, {} "
+ + "merging, {} pairs pending removal, {} segment(s) reapable now, {} of those too "
+ + "little dead to be worth rewriting{}{}), on {}",
+ aligned,
+ mergeSegmentsAtOnce,
+ purged,
+ dead,
+ segmentInfos.size(),
+ compactable.size(),
+ merging.size(),
+ pendingPairRemovals.size(),
+ reapableBySegment.size(),
+ notWorthIt,
+ bestDeclined == null
+ ? ""
+ : String.format(
+ Locale.ROOT,
+ " -- the best of them, %s, offers %d of %d bytes (%d of %d columns) against a"
+ + " %d byte threshold",
+ bestDeclined.info().info.name,
+ bestDeclined.reclaimableBytes(),
+ bestDeclined.sizeInBytes(),
+ bestDeclined.dead().size(),
+ bestDeclined.width(),
+ minReclaimableBytesToPurge),
+ queueUnderPressure ? ", threshold waived: queue over high water" : "",
+ mergeTrigger);
+ }
+ } else if (log.isDebugEnabled()) {
+ log.debug(
+ "AUXIJOIN sidecar: nothing to do on {}, {} segments ({} compactable, {} merging, {} pairs"
+ + " pending removal)",
+ mergeTrigger,
+ segmentInfos.size(),
+ compactable.size(),
+ merging.size(),
+ pendingPairRemovals.size());
+ }
+ return spec;
+ }
+
+ private static Set<String> intersection(Set<String> pairFieldNames, Set<String> reapSnapshot) {
+ if (pairFieldNames.isEmpty() || reapSnapshot.isEmpty()) {
+ return Set.of();
+ }
+ Set<String> both = new HashSet<>();
+ // iterate the smaller side; a fat segment carries a thousand pairs, the queue up to four
+ Set<String> smaller;
+ Set<String> larger;
+ if (pairFieldNames.size() <= reapSnapshot.size()) {
+ smaller = pairFieldNames;
+ larger = reapSnapshot;
+ } else {
+ smaller = reapSnapshot;
+ larger = pairFieldNames;
+ }
+ for (String name : smaller) {
+ if (larger.contains(name)) {
+ both.add(name);
+ }
+ }
+ return both;
+ }
+
+ /**
+ * Whether a {@link DocAlignedMerge} may rewrite this segment at all -- weaker than {@link
+ * #isCompactable}, which additionally declines segments too wide to be worth folding into
+ * another. Those are still worth purging dead columns from.
+ */
+ private static boolean isRewritable(SegmentCommitInfo info) {
+ // no deletions: PaddedToMaxDoc refuses to align a segment carrying them, and the sidecar never
+ // deletes, so this is belt and braces
+ return !info.hasDeletions() && info.info.maxDoc() > 0;
+ }
+
+ /**
+ * Forgets the pair names a merge just dropped the columns of. Called from {@code
+ * OneMerge#mergeFinished} on success only: on failure those columns are still in the index, and
+ * dropping their names here would strand them, never to be reaped again.
+ */
+ private void drainReaped(Set<String> reaped) {
+ pendingPairRemovals.removeAll(reaped);
+ pendingPairRemovalsOrder.removeAll(reaped);
+ reapedPairCount.addAndGet(reaped.size());
+ if (log.isInfoEnabled()) {
+ log.info(
+ "AUXIJOIN sidecar: reaped {} dead pair column(s), {} still pending",
+ reaped.size(),
+ pendingPairRemovals.size());
+ }
+ }
+
+ private static MergeSpecification added(MergeSpecification spec, OneMerge merge) {
+ if (spec == null) {
+ spec = new MergeSpecification();
+ }
+ spec.add(merge);
+ return spec;
+ }
+
+ /**
+ * Whether this segment may be folded into a {@link DocAlignedMerge}: it must carry pairs worth
+ * keeping, must not be so wide already that rewriting it buys nothing, and must have no deletions
+ * -- the sidecar never deletes, and doc-aligning a segment that somehow did would misreport its
+ * live docs for the padded tail.
+ */
+ private boolean isCompactable(SegmentCommitInfo info, Set<String> pairFieldNames) {
+ return !pairFieldNames.isEmpty()
+ && pairFieldNames.size() <= maxPairsPerSegment
+ && !info.hasDeletions()
+ && info.info.maxDoc() > 0;
+ }
+
+ /** The segment's pair field names, from last round's cache when it was already read. */
+ private Set<String> pairFieldNames(SegmentCommitInfo info, Map<String, Set<String>> thisRound)
+ throws IOException {
+ // fieldInfosGen changes whenever the segment's field set can have changed, so it belongs in
+ // the key; segment names are never reused within an index
+ String key = info.info.name + ":" + info.getFieldInfosGen();
+ Set<String> cached = pairFieldNamesBySegment.get(key);
+ if (cached == null) {
+ cached = JoinIndexUtils.pairFieldNames(JoinIndexUtils.readFieldInfos(info));
+ JoinIndexUtils.logDiagnostic(
+ log, "AUXIJOIN evt=readFieldInfos segment={} pairs={}", key, cached.size());
+ }
+ thisRound.put(key, cached);
+ return cached;
+ }
+
+ // counts columns the liveness sweep condemned, as opposed to the snapshot diff; test-only
+ // observability, see strandedColumnCount()
+ private final AtomicInteger strandedColumnCount = new AtomicInteger();
+
+ // counts merges that actually dropped a fully-dead segment; test-only observability, see
+ // droppedSegmentCount()
+ private final AtomicInteger droppedSegmentCount = new AtomicInteger();
+
+ // counts doc-aligned merges that actually produced a compacted segment, see alignedMergeCount()
+ private final AtomicInteger alignedMergeCount = new AtomicInteger();
+
+ // counts single-segment rewrites that existed only to drop dead columns, see purgedSegmentCount()
+ private final AtomicInteger purgedSegmentCount = new AtomicInteger();
+
+ // counts pair names actually reclaimed -- queued as dead and then dropped from the index by a
+ // merge that committed, see reapedPairCount()
+ private final AtomicInteger reapedPairCount = new AtomicInteger();
+
+ /**
+ * A merge over a single dead segment whose contents are reported as fully deleted, so {@link
+ * IndexWriter} drops it instead of rewriting it -- see {@link #wrapForMerge}. Non-static so it
+ * can report back to the outer policy's {@link #droppedSegmentCount}.
+ */
+ private final class DropSegmentMerge extends OneMerge {
+
+ /** The segment's pairs -- all of them dead, which is why it is being dropped. */
+ private final Set<String> reapedPairs;
+
+ DropSegmentMerge(List<SegmentCommitInfo> segments, Set<String> reapedPairs) {
+ super(segments);
+ this.reapedPairs = Set.copyOf(reapedPairs);
+ }
+
+ @Override
+ public CodecReader wrapForMerge(CodecReader reader) {
+ return DocAlignedMerge.dropAllDocs(reader);
+ }
+
+ @Override
+ public void mergeFinished(boolean success, boolean segmentDropped) throws IOException {
+ if (segmentDropped) {
+ int dropped = droppedSegmentCount.incrementAndGet();
+ log.info("AUXIJOIN sidecar: reaped dead segment {} ({} so far)", segments, dropped);
+ // the columns went with the segment, so their queue entries can go too -- otherwise the
+ // pending set only ever grows, which is what exhausted the heap
+ drainReaped(reapedPairs);
+ }
+ super.mergeFinished(success, segmentDropped);
+ }
+ }
+
+ /**
+ * Test-only: how many columns the liveness sweep found stranded -- their side gone, and no sample
+ * able to say so.
+ */
+ int strandedColumnCount() {
+ return strandedColumnCount.get();
+ }
+
+ /** Test-only: how many sidecar segments this policy has actually reaped so far. */
+ int droppedSegmentCount() {
+ return droppedSegmentCount.get();
+ }
+
+ /** Test-only: how many doc-aligned compactions have actually completed so far. */
+ int alignedMergeCount() {
+ return alignedMergeCount.get();
+ }
+
+ /** Test-only: how many segments have been rewritten purely to drop dead pair columns. */
+ int purgedSegmentCount() {
+ return purgedSegmentCount.get();
+ }
+
+ /**
+ * Test-only: how many dead pair names have been fully reclaimed so far -- queued as dead, their
+ * columns dropped by a merge that committed, and their names taken off the queue. Distinct from
+ * {@link #pendingPairRemovalsCount()}, which is what is still owed: that one settles back to zero
+ * as reaping keeps up, so a test wanting to prove reaping happened has to watch this instead.
+ */
+ int reapedPairCount() {
+ return reapedPairCount.get();
+ }
+
+ /** Test-only: how many dead pair field names are currently queued for the next reap. */
+ int pendingPairRemovalsCount() {
+ return pendingPairRemovals.size();
+ }
+
+ /**
+ * Test-only: queues a pair name for reaping, exactly as a sample that stopped needing it would --
+ * so a test can pin down what gets reaped without staging two searcher generations to imply it.
+ */
+ boolean queueForRemoval(String pairFieldName) {
+ return addBounded(
+ pendingPairRemovals, pendingPairRemovalsOrder, pairFieldName, MAX_PENDING_PAIR_REMOVALS);
+ }
+
+ @Override
+ public MergePolicy.MergeSpecification findForcedMerges(
+ SegmentInfos segmentInfos,
+ int maxSegmentCount,
+ Map<SegmentCommitInfo, Boolean> segmentsToMerge,
+ MergeContext mergeContext)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public MergePolicy.MergeSpecification findForcedDeletesMerges(
+ SegmentInfos segmentInfos, MergeContext mergeContext) throws IOException {
+ return null;
+ }
+
+ // caps how many distinct (from-searcher, to-searcher) pairs we remember snapshots for; a
+ // best-effort bound since this only anchors a heuristic reap, never correctness
+ private static final int MAX_TRACKED_SEARCHER_PAIRS = 256;
+
+ /** A sample's needed-pair set, tagged with the index generations it was taken from. */
+ private record NeededPairs(Set<String> pairs, long fromVersion, long toVersion) {}
+
+ /**
+ * {@link DirectoryReader#getVersion()} is unavailable for this reader, so it cannot be ordered.
+ */
+ private static final long UNKNOWN_VERSION = Long.MIN_VALUE;
+
+ private final ConcurrentHashMap<Map.Entry<Object, Object>, NeededPairs>
+ lastNeededPairsBySearcherPair = new ConcurrentHashMap<>();
+ private final ConcurrentLinkedQueue<Map.Entry<Object, Object>> trackedSearcherPairsOrder =
+ new ConcurrentLinkedQueue<>();
+
+ // pair field names seen in an earlier snapshot but missing from a later one for the same
+ // (from-searcher, to-searcher) pair -- i.e. no longer needed -- queued here for findMerges to
+ // reap; also size-capped, same reasoning
+ private static final int MAX_PENDING_PAIR_REMOVALS = 4096;
+
+ /**
+ * Pending removals past which purges ignore {@link #MAX_PENDING_PAIR_REMOVALS}. That threshold
+ * trades heap for I/O -- every name it declines to act on pins a column nobody can read again --
+ * and the trade stops being sensible once the queue is itself what threatens the process. Half
+ * the queue's cap leaves room to work the backlog off before it starts evicting names, which
+ * would strand their columns for good.
+ */
+ private static final int PENDING_PURGE_HIGH_WATER = MAX_PENDING_PAIR_REMOVALS / 2;
+
+ /**
+ * Marks a field name seen on the from side of more than one index; see {@link #onCreateWeight}.
+ */
+ private static final Object AMBIGUOUS_FROM_INDEX = new Object();
+
+ /** The from-side index observed per join field, or {@link #AMBIGUOUS_FROM_INDEX} once several. */
+ private final Map<String, Object> fromIndexByField = new ConcurrentHashMap<>();
+
+ /** Fields already warned about above, so an ambiguous setup doesn't warn on every sample. */
+ private final Set<String> warnedAmbiguousFromFields = ConcurrentHashMap.newKeySet();
+
+ private final Set<String> pendingPairRemovals = ConcurrentHashMap.newKeySet();
+ private final ConcurrentLinkedQueue<String> pendingPairRemovalsOrder =
+ new ConcurrentLinkedQueue<>();
+
+ // how often onCreateWeight actually bothers to sample searcher state; calls arriving sooner
+ // than this after the last accepted sample are skipped outright, since sampling is only a
+ // heuristic hint feeding findMerges' reap decision, not a correctness requirement. Zero (or
+ // negative) disables throttling entirely. Defaults to one minute; see #setSweepInterval.
+ private volatile long samplingIntervalNanos = TimeUnit.MINUTES.toNanos(1);
+
+ // Long.MIN_VALUE marks "never sampled yet" so the very first call always goes through,
+ // regardless of what System.nanoTime()'s arbitrary origin happens to be.
+ private final AtomicLong nextSampleAtNanos = new AtomicLong(Long.MIN_VALUE);
+
+ /**
+ * Configures how often {@link #onCreateWeight} actually samples searcher state for the dead-pair
+ * reaper; calls arriving sooner than this after the last accepted sample are skipped, since
+ * sampling is only a heuristic hint, not a correctness requirement. Defaults to one minute. Pass
+ * zero (or a non-positive value) to sample on every call.
+ */
+ void setSweepInterval(long duration, TimeUnit unit) {
+ this.samplingIntervalNanos = unit.toNanos(duration);
+ }
+
+ /**
+ * Approximates "has enough time passed since the last sample" with {@link System#nanoTime()} --
+ * the JDK's cheapest monotonic timer, since it need not track wall-clock time -- gated by a
+ * single CAS so that under concurrent callers exactly one wins a given interval and the rest
+ * skip, without any lock.
+ */
+ private boolean shouldSample() {
+ long interval = samplingIntervalNanos;
+ if (interval <= 0) {
+ return true;
+ }
+ long now = System.nanoTime();
+ long next = nextSampleAtNanos.get();
+ // subtraction (not direct comparison) so this stays correct across nanoTime() overflow, per
+ // its javadoc
+ if (next != Long.MIN_VALUE && now - next < 0) {
+ return false;
+ }
+ return nextSampleAtNanos.compareAndSet(next, now + interval);
+ }
+
+ void onCreateWeight(
+ Set<String> neededPairs,
+ String fromField,
+ IndexSearcher fromSearcher,
+ String toField,
+ IndexSearcher searcher)
+ throws IOException {
+ if (!shouldSample()) {
+ return;
+ }
+ Object fromKey =
+ JoinIndexUtils.directoryKey(JoinIndexUtils.directory(fromSearcher.getIndexReader()));
+ Object toKey = JoinIndexUtils.directoryKey(JoinIndexUtils.directory(searcher.getIndexReader()));
+ Map.Entry<Object, Object> searcherKey = Map.entry(fromKey, toKey);
+
+ NeededPairs current =
+ new NeededPairs(
+ Set.copyOf(neededPairs), readerVersion(fromSearcher), readerVersion(searcher));
+ NeededPairs previous = lastNeededPairsBySearcherPair.get(searcherKey);
+ // A pair is queued because an earlier sample needed it and this one doesn't. That only means
+ // the pair is dead if this sample sees at least as much of the index as the earlier one did:
+ // the key here is the directory, stable across reopens, so consecutive samples come from
+ // different queries that may hold different searcher generations. Let an older generation
+ // speak and it reports every segment opened since as missing -- queueing live columns for
+ // removal, which JoinIndexScorerSupplier then has to rebuild mid-query.
+ if (previous != null && seesLessThan(current, previous)) {
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=sweepSkipped needed={} wasNeeded={} fromVer={}<{} toVer={}<{}"
+ + " pendingTotal={}",
+ current.pairs().size(),
+ previous.pairs().size(),
+ current.fromVersion(),
+ previous.fromVersion(),
+ current.toVersion(),
+ previous.toVersion(),
+ pendingPairRemovals.size());
+ return;
+ }
+ AuxIndexJoinMergePolicy.putBounded(
+ lastNeededPairsBySearcherPair,
+ trackedSearcherPairsOrder,
+ searcherKey,
+ current,
+ MAX_TRACKED_SEARCHER_PAIRS);
+ int queued = 0;
+ if (previous != null) {
+ for (String pairFieldName : previous.pairs()) {
+ if (!current.pairs().contains(pairFieldName)) {
+ AuxIndexJoinMergePolicy.addBounded(
+ pendingPairRemovals,
+ pendingPairRemovalsOrder,
+ pairFieldName,
+ MAX_PENDING_PAIR_REMOVALS);
+ queued++;
+ }
+ }
+ }
+ int stranded =
+ queueColumnsWithDeadSideKeys(fromField, fromSearcher, fromKey, toField, searcher);
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=sweepSample needed={} wasNeeded={} queuedForRemoval={} strandedFound={}"
+ + " pendingTotal={}",
+ current.pairs().size(),
+ previous == null ? -1 : previous.pairs().size(),
+ queued,
+ stranded,
+ pendingPairRemovals.size());
+ }
+
+ /**
+ * Queues every column the sidecar holds whose side key names a segment that no longer exists --
+ * the deaths the snapshot diff above structurally cannot see.
+ *
+ * <p>That diff only ever reports a pair as dead by watching a query need it and then stop needing
+ * it, both within one process. Everything a restart inherits therefore lives forever: no query in
+ * the new process ever needs a column whose from-segment was merged away while Solr was down, so
+ * no snapshot ever holds its name to miss it later. A 96MB sidecar carrying a 74MB segment from
+ * the previous day's run is what that looks like on disk. Here the reader itself is the evidence
+ * instead: {@code fromSearcher} and {@code searcher} enumerate exactly which side keys still
+ * exist, so any stored pair naming one they don't is provably unreadable, restart or no restart.
+ *
+ * <p>Attribution is by field name, since a sidecar may serve several joins: only pairs whose two
+ * keys belong to this query's {@code fromField} and {@code toField} are judged here, and the
+ * from-side half is judged only while a single from-index has been seen using that field name --
+ * two cores joining into this one on the same field would otherwise each declare the other's
+ * columns dead. The to side needs no such guard: the sidecar lives under the to-core's dataDir,
+ * so every to key in it names a segment of the index {@code searcher} is reading.
+ *
+ * <p>Runs under the same sampling throttle as the snapshot diff, and reads the column names
+ * {@link #findMerges} last saw on disk -- so after a restart the reclaim lands on the first
+ * sample once the sidecar has been committed to at least once, not on the very first query. Like
+ * everything else the reaper decides, this is recoverable rather than authoritative: a column
+ * built moments ago by a concurrent query holding a newer searcher than this sample's is
+ * invisible to these readers and can be condemned, costing a rebuild through {@code
+ * JoinIndexScorerSupplier#refreshJoinTasksReferences} -- never a wrong answer.
+ *
+ * @return how many names this sweep newly queued
+ */
+ private int queueColumnsWithDeadSideKeys(
+ String fromField,
+ IndexSearcher fromSearcher,
+ Object fromDirectoryKey,
+ String toField,
+ IndexSearcher toSearcher) {
+ Map<String, Set<String>> stored = pairFieldNamesBySegment;
+ if (stored.isEmpty()) {
+ return 0;
+ }
+ Set<String> liveFrom = JoinIndexUtils.liveSideKeys(fromSearcher, fromField);
+ Set<String> liveTo = JoinIndexUtils.liveSideKeys(toSearcher, toField);
+ boolean judgeFromSide = fromSideIsUnambiguous(fromField, fromDirectoryKey);
+ int queued = 0;
+ for (Set<String> namesInSegment : stored.values()) {
+ for (String pairFieldName : namesInSegment) {
+ String[] sideKeys = JoinIndexUtils.splitPairFieldName(pairFieldName);
+ if (sideKeys == null
+ || !JoinIndexUtils.isKeyOf(sideKeys[0], fromField)
+ || !JoinIndexUtils.isKeyOf(sideKeys[1], toField)) {
+ continue; // another join's column: this query knows nothing about its sides
+ }
+ boolean toSideGone = !liveTo.contains(sideKeys[1]);
+ boolean fromSideGone = judgeFromSide && !liveFrom.contains(sideKeys[0]);
+ if ((toSideGone || fromSideGone) && queueForRemoval(pairFieldName)) {
+ queued++;
+ strandedColumnCount.incrementAndGet();
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=strandedColumn pair={} fromSideGone={} toSideGone={}",
+ pairFieldName,
+ fromSideGone,
+ toSideGone);
+ }
+ }
+ }
+ return queued;
+ }
+
+ /**
+ * Whether from keys of {@code fromField} can be trusted to come from {@code fromDirectoryKey}'s
+ * index -- true until a second index is seen joining on that same field name, and false from then
+ * on, since neither can then tell which of them wrote a given column.
+ */
+ private boolean fromSideIsUnambiguous(String fromField, Object fromDirectoryKey) {
+ Object seen =
+ fromIndexByField.merge(
+ fromField,
+ fromDirectoryKey,
+ (previous, current) -> previous.equals(current) ? previous : AMBIGUOUS_FROM_INDEX);
+ if (Objects.equals(seen, AMBIGUOUS_FROM_INDEX)) {
+ if (warnedAmbiguousFromFields.add(fromField)) { // once per field, not once per sample
+ log.warn(
+ "AUXIJOIN sidecar: more than one index joins on field {}, so stranded columns of that "
+ + "field can no longer be told apart by their from key and stay until their to "
+ + "side dies",
+ fromField);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Whether {@code sample} was taken from an older view of either side than {@code previous} -- so
+ * a pair it does not need may simply be one it cannot see yet. Unknown versions compare as
+ * neither older nor newer, leaving the sample to be trusted as before.
+ */
+ private static boolean seesLessThan(NeededPairs sample, NeededPairs previous) {
+ return isOlder(sample.fromVersion(), previous.fromVersion())
+ || isOlder(sample.toVersion(), previous.toVersion());
+ }
+
+ private static boolean isOlder(long version, long than) {
+ return version != UNKNOWN_VERSION && than != UNKNOWN_VERSION && version < than;
+ }
+
+ /**
+ * The index generation behind a searcher, or {@link #UNKNOWN_VERSION} when it is not reading a
+ * {@link DirectoryReader} and so carries no orderable version.
+ */
+ private static long readerVersion(IndexSearcher searcher) {
+ return searcher.getIndexReader() instanceof DirectoryReader dr
+ ? dr.getVersion()
+ : UNKNOWN_VERSION;
+ }
+
+ /**
+ * Puts {@code key} -> {@code value}, evicting the oldest key(s) once {@code map} exceeds {@code
+ * maxSize}. Approximate under races (an eviction can drop a key concurrently re-inserted, or the
+ * map can briefly exceed {@code maxSize}) -- acceptable since callers only use this as a soft cap
+ * on a best-effort cache.
+ */
+ static <K, V> V putBounded(
+ ConcurrentHashMap<K, V> map,
+ ConcurrentLinkedQueue<K> insertionOrder,
+ K key,
+ V value,
+ int maxSize) {
+ V previous = map.put(key, value);
+ if (previous == null) {
+ insertionOrder.add(key);
+ while (map.size() > maxSize) {
+ K oldest = insertionOrder.poll();
+ if (oldest == null) {
+ break;
+ }
+ map.remove(oldest);
+ }
+ }
+ return previous;
+ }
+
+ /**
+ * Same eviction policy as {@link AuxIndexJoinMergePolicy#putBounded}, for a plain set.
+ *
+ * @return whether the value was not already there
+ */
+ static <T> boolean addBounded(
+ Set<T> set, ConcurrentLinkedQueue<T> insertionOrder, T value, int maxSize) {
+ if (!set.add(value)) {
+ return false;
+ }
+ insertionOrder.add(value);
+ while (set.size() > maxSize) {
+ T oldest = insertionOrder.poll();
+ if (oldest == null) {
+ break;
+ }
+ set.remove(oldest);
+ }
+ return true;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinQuery.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinQuery.java
new file mode 100644
index 00000000000..4cc1460f1d9
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinQuery.java
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.function.Predicate;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.internal.hppc.IntHashSet;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryVisitor;
+import org.apache.lucene.search.ScoreMode;
+import org.apache.lucene.search.Weight;
+import org.apache.solr.common.util.CollectionUtil;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.JoinSegmentReference;
+import org.jspecify.annotations.NonNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Joins the from-side index to the to-side index this query is executed against, resolving
+ * from-side docs matching {@code fromQuery} to to-side docs through the auxiliary join index
+ * managed by {@link AuxIndexManager}: there, each (from-segment, to-segment) pair owns a
+ * SORTED_NUMERIC column named by both sides' persistent keys, whose doc number is the from-side doc
+ * id and whose value is the matching to-side doc id. Pair columns missing from the join index are
+ * built on demand at weight creation, so no explicit build step exists; obtain instances via {@link
+ * AuxIndexManager#newJoinQuery}. Matches score a constant.
+ *
+ * @lucene.experimental
+ */
+class AuxIndexJoinQuery extends Query {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final AuxIndexManager joinIndex;
+ final String fromField;
+ final Query fromQuery;
+ protected final IndexSearcher fromSearcher;
+ final String toField;
+ private final ExecutorService fromExecutorService;
+
+ AuxIndexJoinQuery(
+ AuxIndexManager joinIndex,
+ String fromField,
+ Query fromQuery,
+ IndexSearcher fromSearcher,
+ String toField,
+ ExecutorService fromExecutorService) {
+ this.joinIndex = Objects.requireNonNull(joinIndex, "joinIndex");
+ this.fromField = Objects.requireNonNull(fromField, "fromField");
+ this.fromQuery = Objects.requireNonNull(fromQuery, "fromQuery");
+ this.fromSearcher = Objects.requireNonNull(fromSearcher, "fromSearcher");
+ this.toField = Objects.requireNonNull(toField, "toField");
+ this.fromExecutorService = fromExecutorService;
+ }
+
+ @SuppressWarnings("ReferenceEquality")
+ @Override
+ public Query rewrite(IndexSearcher indexSearcher) throws IOException {
+ // the from-side selection rewrites against the from-side searcher, not against the (to-side)
+ // searcher this query is executed with
+ Query rewrittenFrom = fromQuery.rewrite(fromSearcher);
+ if (rewrittenFrom != fromQuery) { // TODO check MatchNoDocs ?
+ return new AuxIndexJoinQuery(
+ joinIndex, fromField, rewrittenFrom, fromSearcher, toField, fromExecutorService);
+ }
+ return super.rewrite(indexSearcher);
+ }
+
+ @Override
+ public Weight createWeight(IndexSearcher toSideSearcher, ScoreMode scoreMode, float boost)
+ throws IOException {
+ @NonNull Map<String, AuxIndexManager.SegmentsTuple> neededPairs =
+ getRequiredColumNames(toSideSearcher);
+
+ joinIndex.onCreateWeight(
+ neededPairs.keySet(), fromField, fromSearcher, toField, toSideSearcher);
+ Predicate<String> isNeeded = neededPairs::containsKey;
+
+ Map<String, JoinSegmentReference> existingJoinSegments;
+ IndexSearcher joinSearcher = this.joinIndex.acquire();
+ try {
+ existingJoinSegments = JoinIndexUtils.extractExistingJoinColumns(joinSearcher, isNeeded);
+ } finally {
+ this.joinIndex.release(joinSearcher);
+ }
+ int pairsNeeded = neededPairs.size();
+ neededPairs.keySet().removeAll(existingJoinSegments.keySet());
+ IntHashSet fromOrdsToLoad = new IntHashSet(neededPairs.size());
+ neededPairs.values().stream()
+ .mapToInt(AuxIndexManager.SegmentsTuple::fromLeafOrd)
+ .forEach(fromOrdsToLoad::add);
+ if (JoinIndexUtils.diagnosticsEnabled(log)) {
+ // pairsMissing > 0 on a repeat query means those pairs were never persisted by a previous
+ // run (writeBatch never captured them), so their from-segments' FK columns get reloaded
+ // here; pairsClaimed counts missing pairs another thread is building right now (claims are
+ // dropped once persisted), i.e. reloads that are pure waste
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=weight pairsNeeded={} pairsExisting={} pairsMissing={} pairsClaimed={}"
+ + " fkOrdsToLoad={} missingPairs={}",
+ pairsNeeded,
+ existingJoinSegments.size(),
+ neededPairs.size(),
+ joinIndex.countClaimedBuilds(neededPairs.keySet()),
+ fromOrdsToLoad.size(),
+ neededPairs.keySet());
+ }
+ Map<Integer, Future<FromLeafJoinContext>> fromFutures = loadFromSide(fromOrdsToLoad);
+ // TODO this might produce too many small tasks
+ return new JoinIndexWeight(
+ this,
+ joinSearcher,
+ existingJoinSegments,
+ toSideSearcher.getIndexReader(),
+ scoreMode,
+ boost,
+ fromFutures);
+ }
+
+ @SuppressWarnings("unchecked")
+ private Map<Integer, Future<FromLeafJoinContext>> loadFromSide(IntHashSet fromLeafsToLoad)
+ throws IOException {
+ LinkedHashMap<Integer, Future<FromLeafJoinContext>> futuresByLeafOrds =
+ CollectionUtil.newLinkedHashMap(this.fromSearcher.getLeafContexts().size());
+ final Weight fromWeight =
+ this.fromSearcher.createWeight(this.fromQuery, ScoreMode.COMPLETE_NO_SCORES, 1.0f);
+
+ List<LeafReaderContext> fromLeafs = new ArrayList<>(this.fromSearcher.getLeafContexts());
+ // heaviest first, the smallest last
+ fromLeafs.sort(
+ Comparator.<LeafReaderContext, Boolean>comparing(ctx -> fromLeafsToLoad.contains(ctx.ord))
+ .thenComparingInt(ctx -> ctx.reader().maxDoc())
+ .reversed());
+
+ for (LeafReaderContext ctx : fromLeafs) {
+ futuresByLeafOrds.putLast(
+ ctx.ord, // the heaviest is submitted first, and accessed first as well
+ this.fromExecutorService.submit(
+ () ->
+ FromLeafJoinContext.heavyLoadFromLeaf(
+ fromWeight, fromField, ctx, fromLeafsToLoad.contains(ctx.ord))));
+ }
+
+ return futuresByLeafOrds;
+ }
+
+ private @NonNull Map<String, AuxIndexManager.SegmentsTuple> getRequiredColumNames(
+ IndexSearcher searcher) {
+ Map<String, AuxIndexManager.SegmentsTuple> neededPairs = new HashMap<>();
+ for (LeafReaderContext toContext : searcher.getIndexReader().leaves()) {
+ String toKey = JoinIndexUtils.getSideKey(toContext, toField);
+ for (LeafReaderContext fromCtx : fromSearcher.getLeafContexts()) {
+ String fromKey = JoinIndexUtils.getSideKey(fromCtx, fromField);
+ neededPairs.put(
+ fromKey + "_" + toKey, new AuxIndexManager.SegmentsTuple(fromCtx.ord, toContext.ord));
+ }
+ }
+ return neededPairs;
+ }
+
+ @Override
+ public String toString(String field) {
+ return "AuxIndexJoinQuery(" + fromField + " -> " + toField + ", from: " + fromQuery + ")";
+ }
+
+ @Override
+ public void visit(QueryVisitor visitor) {
+ visitor.visitLeaf(this);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return sameClassAs(other) && equalsTo((AuxIndexJoinQuery) other);
+ }
+
+ private boolean equalsTo(AuxIndexJoinQuery other) {
+ // the join index and the from searcher compare by identity: a reopened from reader sees
+ // different ordinal spaces, so queries over different searcher instances must not be
+ // considered equal
+ return joinIndex == other.joinIndex
+ && Objects.equals(fromSearcher, other.fromSearcher)
+ && fromField.equals(other.fromField)
+ && fromQuery.equals(other.fromQuery)
+ && toField.equals(other.toField);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(
+ classHash(),
+ System.identityHashCode(joinIndex),
+ fromField,
+ fromQuery,
+ System.identityHashCode(fromSearcher),
+ toField);
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexManager.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexManager.java
new file mode 100644
index 00000000000..e4e822a56d6
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/AuxIndexManager.java
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.lucene.index.ConcurrentMergeScheduler;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
+import org.apache.lucene.index.IndexNotFoundException;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.MergeScheduler;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.SearcherManager;
+import org.apache.lucene.store.AlreadyClosedException;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.IOUtils;
+import org.apache.solr.client.api.util.SolrVersion;
+import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The auxiliary join index: a self-maintaining sidecar persisting per (from-segment, to-segment)
+ * doc id mappings, so query-time joining reduces to bitset translation. It owns the sidecar's
+ * {@link IndexWriter} and {@link SearcherManager}; pair columns are built lazily when an {@link
+ * AuxIndexJoinQuery} first needs them, so users only construct an instance once, create queries
+ * with {@link #newJoinQuery} and search them with a bare to-side {@link IndexSearcher}:
+ *
+ * <pre class="prettyprint">
+ * AuxIndexManager joinIndex = new AuxIndexManager(joinDir); // once per process
+ * Query q = joinIndex.newJoinQuery(fromField, fromQuery, fromSearcher, toField);
+ * TopDocs hits = toSearcher.search(q, 10);
+ * ...
+ * joinIndex.close(); // app shutdown
+ * </pre>
+ *
+ * <p>After either side reopens, the next query builds only the missing (from, to) segment pairs:
+ * pair columns are addressed by both sides' persistent segment keys, which survive reopens. Pair
+ * columns orphaned by merges are reclaimed by the sidecar's own merge policy, which drops them from
+ * the segments it rewrites; see package's javadoc.
+ *
+ * @lucene.experimental
+ * <p>This is experimental API and subject to change.
+ */
+public final class AuxIndexManager implements Closeable {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private final IndexWriter writer;
+ private final SearcherManager manager;
+ private final JoinColumnIndexer pairBuilder = new JoinColumnIndexer(this);
+
+ // package-private (not private): tests reach in directly to observe the reaper's state
+ final AuxIndexJoinMergePolicy mergePolicy;
+ private final MergeScheduler mergeScheduler;
+ private final JoinColumWriter writerDelegate;
+ private final boolean blockingRefresh;
+ private final boolean useFromSideThreads;
+ private final boolean wipeOnVersionMismatch;
+
+ /**
+ * Commits the sidecar on a timer, or {@code null} when {@link
+ * AuxIndexJoinConfig#setCommitIntervalMs} asked for a commit per batch instead. Deliberately not
+ * holding this manager's monitor: {@link #writeBatch} is synchronized only to keep a batch's
+ * documents contiguous in one segment, which a commit doesn't disturb, so committing off that
+ * lock keeps the fsync out of the builders' way.
+ */
+ private final ScheduledExecutorService committer;
+
+ /** Whether {@link #writeBatch} commits before it returns, per the interval being zero. */
+ private final boolean commitPerBatch;
+
+ /** A pair's (from-segment, to-segment) leaf ordinals. */
+ record SegmentsTuple(int fromLeafOrd, int toLeafOrd) {}
+
+ /**
+ * A pair column's address in the join index: the pair field name and the sidecar segment (name
+ * plus current leaf ordinal) carrying it -- enough to locate and open the column's real
+ * docvalues, or to check whether a pair already exists before deciding what still needs to be
+ * built. A resolved cell's edges are tracked separately, as a plain {@code DocEdges}, since they
+ * don't change as this reference is refreshed.
+ */
+ record JoinSegmentReference(
+ String pairFieldName, String joinSegmentName, int joinSegmentLeafOrd) {}
+
+ /**
+ * Opens a persistent auxiliary join index over the given directory, creating it if empty, using
+ * the default {@link AuxIndexJoinConfig}. The caller retains ownership of the directory: {@link
+ * #close()} does not close it.
+ */
+ public AuxIndexManager(Directory directory) throws IOException {
+ this(directory, new AuxIndexJoinConfig());
+ }
+
+ /**
+ * Opens a persistent auxiliary join index over the given directory, creating it if empty, using
+ * the given {@link AuxIndexJoinConfig}. The caller retains ownership of the directory: {@link
+ * #close()} does not close it.
+ */
+ public AuxIndexManager(Directory directory, AuxIndexJoinConfig config) throws IOException {
+ this(directory, config, new ConcurrentMergeScheduler());
+ }
+
+ /**
+ * Opens a persistent auxiliary join index over the given directory, creating it if empty, using
+ * the given {@link AuxIndexJoinConfig} and {@link MergeScheduler} in place of the default {@link
+ * ConcurrentMergeScheduler}. The caller retains ownership of the directory: {@link #close()} does
+ * not close it.
+ */
+ public AuxIndexManager(
+ Directory directory, AuxIndexJoinConfig config, MergeScheduler mergeScheduler)
+ throws IOException {
+ this.mergeScheduler = mergeScheduler;
+ this.mergePolicy = new AuxIndexJoinMergePolicy();
+ this.mergePolicy.setSweepInterval(config.getSweepSamplingIntervalNanos(), TimeUnit.NANOSECONDS);
+ this.mergePolicy.setCompaction(config.getMergeSegmentsAtOnce(), config.getMaxPairsPerSegment());
+ this.mergePolicy.setMinReclaimableBytesToPurge(config.getMinReclaimableBytesToPurge());
+ this.wipeOnVersionMismatch = config.getWipeOnVersionMismatch();
+ this.writer =
+ new IndexWriter(
+ directory,
+ new IndexWriterConfig().setMergePolicy(mergePolicy).setMergeScheduler(mergeScheduler));
+ // set once, not per batch: live commit data sticks on the writer and is applied by whichever
+ // commit comes next, including the wipe's just below and the one close() makes
+ this.writer.setLiveCommitData(
+ Map.of(JoinIndexUtils.AUX_INDEX_VERSION, SolrVersion.LATEST_STRING).entrySet());
+ wipeIfIncompatibleVersion(directory);
+ this.manager = new SearcherManager(writer, null);
+ JoinColumWriter bulkWriter = new JoinColumnDocWriter(); // new AIJoinColumnWriter()
+ this.writerDelegate =
+ config.getSingleFieldPerSegment()
+ ? new SingleColumnBySegmentWriter(bulkWriter)
+ : bulkWriter;
+ this.blockingRefresh = config.getBlockingRefresh();
+ this.useFromSideThreads = config.getUseFromSideThreads();
+ long commitIntervalMs = config.getCommitIntervalMs();
+ this.commitPerBatch = commitIntervalMs == 0;
+ if (commitPerBatch) {
+ this.committer = null;
+ } else {
+ this.committer =
+ Executors.newSingleThreadScheduledExecutor(
+ new SolrNamedThreadFactory("auxIndexJoinCommit"));
+ this.committer.scheduleWithFixedDelay(
+ this::commitIfChanged, commitIntervalMs, commitIntervalMs, TimeUnit.MILLISECONDS);
+ }
+ }
+
+ /**
+ * Moves the commit point forward if a batch has been flushed since the last one, so Lucene can
+ * let go of the files only the previous commit still referenced. Swallows what it cannot act on:
+ * a failed commit leaves the sidecar exactly where it was, which the next attempt (or the next
+ * query, rebuilding a pair) recovers from on its own -- there is nothing here worth failing the
+ * node over.
+ */
+ private void commitIfChanged() {
+ try {
+ if (writer.hasUncommittedChanges()) {
+ writer.commit();
+ }
+ } catch (AlreadyClosedException e) {
+ // raced close(); the writer commits itself on the way out
+ } catch (Throwable t) {
+ log.warn("AUXIJOIN sidecar: periodic commit failed, retrying at the next interval", t);
+ }
+ }
+
+ /**
+ * Discards the on-disk aux join index when it was written by a Solr whose major version differs
+ * from the current one: doc-id mappings are version-specific, so a mismatched index must not be
+ * reused and its pairs get rebuilt lazily. The version is read from the last commit's user data,
+ * written by {@link JoinColumnDocWriter}; a missing or unparseable version is treated as
+ * incompatible. Runs before the {@link SearcherManager} opens so the fresh reader sees a clean
+ * index. No-op when {@link AuxIndexJoinConfig#setWipeOnVersionMismatch} is {@code false}.
+ */
+ private void wipeIfIncompatibleVersion(Directory directory) throws IOException {
+ if (!wipeOnVersionMismatch) {
+ return;
+ }
+ List<IndexCommit> commits;
+ try {
+ commits = DirectoryReader.listCommits(directory);
+ } catch (IndexNotFoundException e) {
+ return; // fresh index, nothing committed yet to check
+ }
+ if (commits.isEmpty()) {
+ return;
+ }
+ String committedVersion =
+ commits.get(commits.size() - 1).getUserData().get(JoinIndexUtils.AUX_INDEX_VERSION);
+ boolean incompatible;
+ try {
+ incompatible =
+ committedVersion == null
+ || SolrVersion.valueOf(committedVersion).getMajorVersion()
+ != SolrVersion.LATEST.getMajorVersion();
+ } catch (RuntimeException e) { // unparseable committed version
+ incompatible = true;
+ }
+ if (incompatible) {
+ writer.deleteAll();
+ writer.commit();
+ }
+ }
+
+ /**
+ * Creates a query joining the docs matching {@code fromQuery} in {@code fromSearcher}'s index to
+ * the index the returned query is executed against, through {@code fromField} = {@code toField}
+ * term equality. Missing pair columns are built into this join index on first execution.
+ *
+ * @deprecated use another constructor passing executor service
+ */
+ @Deprecated
+ public Query newJoinQuery(
+ String fromField, Query fromQuery, IndexSearcher fromSearcher, String toField) {
+ return newJoinQuery(fromField, fromQuery, fromSearcher, toField, new DirectExecutorService());
+ }
+
+ public Query newJoinQuery(
+ String fromField,
+ Query fromQuery,
+ IndexSearcher fromSearcher,
+ String toField,
+ ExecutorService fromExecutor) {
+ if (fromExecutor == null || !useFromSideThreads) {
+ fromExecutor = new DirectExecutorService();
+ }
+ return new AuxIndexJoinQuery(this, fromField, fromQuery, fromSearcher, toField, fromExecutor);
+ }
+
+ /**
+ * How many of the given pair field names have a build currently in flight (claims are removed
+ * once persisted). Diagnostic only; see {@link JoinColumnIndexer#countClaimedBuilds}.
+ */
+ int countClaimedBuilds(Set<String> pairFieldNames) {
+ return pairBuilder.countClaimedBuilds(pairFieldNames);
+ }
+
+ IndexSearcher acquire() throws IOException {
+ return manager.acquire();
+ }
+
+ void release(IndexSearcher searcher) throws IOException {
+ manager.release(searcher);
+ }
+
+ /**
+ * Builds and persists the given missing pair columns, keyed by pair field name to their
+ * (from-segment, to-segment) leaf ordinals. Delegates to {@link
+ * JoinColumnIndexer#buildAndPersistJoinColumns}, which documents the claim/await dedup and the
+ * fresh-searcher double-check in detail.
+ *
+ * @param observedAbsentSearcher the join-index searcher in which the caller established that
+ * {@code missingPairs} are absent; possibly stale by now. Pass {@code null} when absence
+ * wasn't verified against a live searcher -- that forces the re-check scan.
+ * @return in memory data for just written segemts
+ */
+ Map<String, JoinColumnModel> buildAndPersistJoinColumns(
+ Map<String, SegmentsTuple> missingPairs,
+ IndexReader fromReader,
+ IndexReader toReader,
+ String toField,
+ String traceCtxId,
+ IndexSearcher observedAbsentSearcher,
+ Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures)
+ throws IOException, ExecutionException, InterruptedException {
+ return pairBuilder.buildAndPersistJoinColumns(
+ missingPairs,
+ fromReader,
+ toReader,
+ toField,
+ traceCtxId,
+ observedAbsentSearcher,
+ fromColumnFutures);
+ }
+
+ /**
+ * Serializes sidecar writes: one batch per commit keeps every batch at doc 0 of its own segment,
+ * preserving pair-column doc number == from-side doc id. Builders' futures are completed before
+ * this runs, so waiters consume the in-memory models without paying for the commit and refresh
+ * here -- a completed future does not mean the reader already exposes the column.
+ *
+ * <p>It should be plain simple synchronized. As alternatives
+ *
+ * <ul>
+ * <li>push synchronized deeper to iw.addDocs(),iw.commit()
+ * <li>merge columns from concurrent threads and writes them as a single batch - too much </>
+ * The invariant is a column starts ad doc#==0, this iw.addDocs(),iw.commit() goes one by
+ * one. But such segment might contain many parallel columns, since they have distinguishing
+ * names.
+ */
+ synchronized void writeBatch(Map<String, JoinColumnModel> mappings) throws IOException {
+ this.writerDelegate.writeJoinColumns(writer, mappings);
+ if (this.commitPerBatch) {
+ writer.commit();
+ }
+ if (this.blockingRefresh) {
+ manager.maybeRefreshBlocking();
+ } else {
+ manager.maybeRefresh(); // perhaps it should be carried out the enclosing synchronize
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (committer != null) {
+ // stop the timer before the writer goes, so a commit can't be mid-flight when it does
+ ExecutorUtil.shutdownAndAwaitTermination(committer);
+ }
+ IOUtils.close(manager, writer); // IndexWriter.close() commits what the timer hasn't
+ }
+
+ public void onCreateWeight(
+ Set<String> neededPairs,
+ String fromField,
+ IndexSearcher fromSearcher,
+ String toField,
+ IndexSearcher searcher)
+ throws IOException {
+ this.mergePolicy.onCreateWeight(neededPairs, fromField, fromSearcher, toField, searcher);
+ }
+
+ /**
+ * Test-only: blocks until any in-flight merges (e.g. a dead-pair reap) finish. Only supported
+ * when this index was opened with a {@link ConcurrentMergeScheduler} (the default).
+ */
+ void waitForMerges() {
+ if (mergeScheduler instanceof ConcurrentMergeScheduler cms) {
+ cms.sync();
+ } else {
+ throw new UnsupportedOperationException(
+ "waitForMerges() requires a ConcurrentMergeScheduler, got " + mergeScheduler.getClass());
+ }
+ }
+
+ private static class SingleColumnBySegmentWriter extends JoinColumWriter {
+ private final JoinColumWriter bulkWriter;
+
+ public SingleColumnBySegmentWriter(JoinColumWriter bulkWriter) {
+ this.bulkWriter = bulkWriter;
+ }
+
+ @Override
+ void writeJoinColumns(IndexWriter writer, Map<String, JoinColumnModel> mappings)
+ throws IOException {
+ for (Map.Entry<String, JoinColumnModel> entry : mappings.entrySet()) {
+ bulkWriter.writeJoinColumns(writer, Map.of(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DirectExecutorService.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DirectExecutorService.java
new file mode 100644
index 00000000000..050bde94019
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DirectExecutorService.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.util.List;
+import java.util.concurrent.AbstractExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * An {@link java.util.concurrent.ExecutorService} that runs every submitted task immediately on the
+ * calling thread, preserving the ordering and stack of the submitter. Useful to make asynchronous
+ * join work deterministic (e.g. in tests) or when the caller wants no background concurrency.
+ *
+ * @lucene.experimental
+ */
+final class DirectExecutorService extends AbstractExecutorService {
+
+ private volatile boolean shutdown;
+
+ @Override
+ public void shutdown() {
+ shutdown = true;
+ }
+
+ @Override
+ public List<Runnable> shutdownNow() {
+ shutdown = true;
+ return List.of();
+ }
+
+ @Override
+ public boolean isShutdown() {
+ return shutdown;
+ }
+
+ @Override
+ public boolean isTerminated() {
+ return shutdown;
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit unit) {
+ return shutdown;
+ }
+
+ @Override
+ public void execute(Runnable command) {
+ command.run();
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DocAlignedMerge.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DocAlignedMerge.java
new file mode 100644
index 00000000000..9e5b41cd63d
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/DocAlignedMerge.java
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.FilterCodecReader;
+import org.apache.lucene.index.FilterLeafReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.MergePolicy;
+import org.apache.lucene.index.ParallelLeafReader;
+import org.apache.lucene.index.SegmentCommitInfo;
+import org.apache.lucene.index.SlowCodecReaderWrapper;
+import org.apache.lucene.index.StoredFieldVisitor;
+import org.apache.lucene.index.StoredFields;
+import org.apache.lucene.index.TermVectors;
+import org.apache.lucene.util.Bits;
+
+/**
+ * Compacts sidecar segments <em>without moving a single doc id</em>: doc {@code i} of the merged
+ * segment carries the columns of doc {@code i} of every input segment, and the result's {@code
+ * maxDoc} is the largest input's, not the sum.
+ *
+ * <pre>
+ * segment _a (maxDoc 2) segment _b (maxDoc 3) merged (maxDoc 3)
+ * doc0: A B doc0: D E F doc0: A B D E F
+ * doc1: A B doc1: D E F doc1: A B D E F
+ * doc2: D E F doc2: D E F
+ * </pre>
+ *
+ * <p><b>Why not an ordinary merge.</b> Lucene merges by concatenating, so segment _b's doc 0 would
+ * become doc 2 of the result. A sidecar doc id is not an identity of its own -- it <em>is</em> the
+ * from-side doc id the column is addressed by ({@link JoinIndexScorerSupplier} reads a pair with
+ * {@code toDocsByFromDoc.advanceExact(fromDoc)}), and every pair's edges live at doc 0 -- so
+ * concatenation silently shifts every mapping. That is why {@link AuxIndexJoinMergePolicy} could
+ * only ever drop whole dead segments and never rewrite live ones, and why the sidecar grew by one
+ * segment per written batch until the process ran out of mmap-able address space.
+ *
+ * <p>Aligning is well defined precisely because this merge is <em>opaque</em>: it never looks
+ * inside a column. A pair field name carries both sides' segment ids ({@link
+ * JoinIndexUtils#pairFieldName}), so columns of different pairs never collide, and a doc id means
+ * the same thing in every input -- the from-side doc id of that column's own from-segment. Inputs
+ * built from a smaller from-segment simply have no values in the tail docs.
+ *
+ * <p><b>How it hooks into Lucene.</b> {@link IndexWriter} merges by concatenating whatever {@link
+ * #wrapForMerge} hands back, so this merge hands back nothing for all inputs but one: every input
+ * except the last is wrapped as fully deleted ({@link org.apache.lucene.util.Bits.MatchNoBits},
+ * {@code numDocs() == 0}) so it contributes no doc, and the last is replaced by a {@link
+ * ParallelLeafReader} over <em>all</em> the inputs -- the reader that aligns fields by doc id.
+ * Concatenating {@code [nothing, ..., nothing, union]} is the union. This relies on {@code
+ * IndexWriter#mergeMiddle} calling {@code wrapForMerge} exactly once per segment, in {@link
+ * #segments} order, and only then starting the merge, so that by the last call every input reader
+ * is known; the count is enforced below.
+ *
+ * <p><b>Reaping rides along.</b> The columns of pairs queued for removal are dropped while the
+ * segment is being rewritten anyway: {@link #wrapForMerge} hides their fields from every input, so
+ * the merged segment never gets them. This is what actually reclaims a dead pair -- dropping a
+ * whole segment instead needs <em>all</em> of its pairs to be dead at once, which a segment
+ * carrying hundreds of them never manages, and the pending set then grows without bound. A pair
+ * that vanishes under a query still holding a reference to it is rebuilt by {@code
+ * JoinIndexScorerSupplier#refreshJoinTasksReferences}, which is what makes dropping columns safe;
+ * before that it failed the query outright.
+ *
+ * <p>{@code reapedPairs} is a snapshot taken when the merge is created, not the live set: a pair
+ * queued while this merge runs is not dropped by it and must stay queued for the next one. {@link
+ * #mergeFinished} drains exactly this snapshot, and only on success -- on failure the columns are
+ * still there.
+ *
+ * @lucene.experimental
+ */
+final class DocAlignedMerge extends MergePolicy.OneMerge {
+
+ /** The readers handed to {@link #wrapForMerge}, in {@link #segments} order. */
+ private final List<CodecReader> inputs;
+
+ private final Runnable onMerged;
+
+ /** Pair names whose columns this merge drops; handed to {@link #onReaped} once it commits. */
+ private final Set<String> reapedPairs;
+
+ /** The fields those pairs occupy, precomputed for the per-field test in {@link #hideReaped}. */
+ private final Set<String> reapedFields;
+
+ private final Consumer<Set<String>> onReaped;
+ private ParallelLeafReader union;
+
+ DocAlignedMerge(
+ List<SegmentCommitInfo> segments,
+ Set<String> reapedPairs,
+ Runnable onMerged,
+ Consumer<Set<String>> onReaped) {
+ super(segments);
+ if (segments.size() < 2 && reapedPairs.isEmpty()) {
+ // one input and nothing to drop would rewrite it byte for byte; one input with columns to
+ // drop is a purge, and a single input is trivially doc-aligned with itself
+ throw new IllegalArgumentException(
+ "a doc-aligned merge of " + segments.size() + " segment(s) would only rewrite it");
+ }
+ this.inputs = new ArrayList<>(segments.size());
+ this.onMerged = onMerged;
+ this.reapedPairs = Set.copyOf(reapedPairs);
+ Set<String> fields = new HashSet<>();
+ for (String pairFieldName : this.reapedPairs) {
+ fields.addAll(JoinIndexUtils.columnFieldNames(pairFieldName));
+ }
+ this.reapedFields = fields;
+ this.onReaped = onReaped;
+ }
+
+ @Override
+ public CodecReader wrapForMerge(CodecReader reader) throws IOException {
+ if (inputs.size() >= segments.size()) {
+ throw new IllegalStateException(
+ "wrapForMerge called "
+ + (inputs.size() + 1)
+ + " times for a merge of "
+ + segments.size()
+ + " segments; the doc-aligned merge builds its union on the last call");
+ }
+ // filter first, drop second: a fully-deleted reader still hands its FieldInfos to the merge, so
+ // a reaped field left visible here would survive into the result as an empty column -- still
+ // named, so still counted as a live pair by JoinIndexUtils#pairFieldNames
+ CodecReader filtered = hideReaped(reader);
+ inputs.add(filtered);
+ if (inputs.size() < segments.size()) {
+ return dropAllDocs(filtered); // contributes nothing; its columns come from the union below
+ }
+ int alignedMaxDoc = 0;
+ for (CodecReader input : inputs) {
+ alignedMaxDoc = Math.max(alignedMaxDoc, input.maxDoc());
+ }
+ LeafReader[] aligned = new LeafReader[inputs.size()];
+ for (int i = 0; i < aligned.length; i++) {
+ aligned[i] = new PaddedToMaxDoc(inputs.get(i), alignedMaxDoc);
+ }
+ // closeSubReaders=false: the readers are IndexWriter's, this merge must not close them
+ union = new ParallelLeafReader(false, aligned);
+ return SlowCodecReaderWrapper.wrap(union);
+ }
+
+ @Override
+ public void mergeFinished(boolean success, boolean segmentDropped) throws IOException {
+ try {
+ if (union != null) {
+ union.close(); // only decrefs the padding views, see PaddedToMaxDoc#doClose
+ union = null;
+ }
+ inputs.clear();
+ if (success && !segmentDropped) {
+ onMerged.run();
+ if (!reapedPairs.isEmpty()) {
+ // the columns are gone from the index now, so the queue entries that named them can go
+ // too -- exactly this snapshot, never the live set
+ onReaped.accept(reapedPairs);
+ }
+ }
+ } finally {
+ super.mergeFinished(success, segmentDropped);
+ }
+ }
+
+ /**
+ * A view of {@code reader} without the columns of any reaped pair -- both hidden from {@link
+ * org.apache.lucene.index.FieldInfos} and answered as absent, so neither the merged {@code
+ * FieldInfos} nor the merged docvalues carry them. Returns the reader untouched when there is
+ * nothing to reap, which is the common case.
+ *
+ * <p>Hiding the field from {@code FieldInfos} is the whole mechanism, and it is enough on its
+ * own: {@code CodecReader#getSortedNumericDocValues} resolves the name through {@code
+ * getFieldInfos()} and answers null when it is not there, {@link ParallelLeafReader} builds its
+ * field-to-reader map from them, and {@code SegmentMerger} merges them to decide which columns
+ * the result has. The underlying producer is left alone -- nothing reaches it for a field no
+ * longer named.
+ */
+ private CodecReader hideReaped(CodecReader reader) {
+ if (reapedFields.isEmpty()) {
+ return reader;
+ }
+ List<FieldInfo> kept = new ArrayList<>();
+ for (FieldInfo fieldInfo : reader.getFieldInfos()) {
+ if (!reapedFields.contains(fieldInfo.name)) {
+ kept.add(fieldInfo);
+ }
+ }
+ if (kept.size() == reader.getFieldInfos().size()) {
+ return reader; // this input carries none of them
+ }
+ FieldInfos survivors = new FieldInfos(kept.toArray(FieldInfo[]::new));
+ return new FilterCodecReader(reader) {
+ @Override
+ public FieldInfos getFieldInfos() {
+ return survivors;
+ }
+
+ @Override
+ public CacheHelper getCoreCacheHelper() {
+ return null; // the field set differs from the wrapped reader's
+ }
+
+ @Override
+ public CacheHelper getReaderCacheHelper() {
+ return null;
+ }
+ };
+ }
+
+ /**
+ * A view of {@code reader} reporting all its docs deleted, so {@link IndexWriter} carries none of
+ * them into the merged segment. Shared with {@link AuxIndexJoinMergePolicy}'s reaper, which drops
+ * a whole dead segment by wrapping it this way and letting the merge produce nothing at all.
+ */
+ static CodecReader dropAllDocs(CodecReader reader) {
+ return new FilterCodecReader(reader) {
+ @Override
+ public CacheHelper getCoreCacheHelper() {
+ return reader.getCoreCacheHelper();
+ }
+
+ @Override
+ public CacheHelper getReaderCacheHelper() {
+ return null; // we are altering live docs
+ }
+
+ @Override
+ public Bits getLiveDocs() {
+ return new Bits.MatchNoBits(reader.maxDoc());
+ }
+
+ @Override
+ public int numDocs() {
+ return 0;
+ }
+ };
+ }
+
+ /**
+ * Presents a sidecar segment as if it had {@code maxDoc} docs, the ones past its own {@code
+ * maxDoc} simply carrying no value in any of its columns -- what {@link ParallelLeafReader}
+ * needs, since it combines readers of equal length only.
+ *
+ * <p>Padding is safe here because a sidecar segment holds nothing but sparse SORTED_NUMERIC
+ * docvalues: their iterators end at the last doc that has a value, well before {@code maxDoc},
+ * exactly as they would for any sparse column. Stored fields and term vectors, which would
+ * otherwise be asked for docs the delegate doesn't have, are answered as empty -- the sidecar
+ * writes neither.
+ */
+ private static final class PaddedToMaxDoc extends FilterLeafReader {
+
+ private static final StoredFields NO_STORED_FIELDS =
+ new StoredFields() {
+ @Override
+ public void document(int docID, StoredFieldVisitor visitor) {}
+ };
+
+ private final int maxDoc;
+
+ PaddedToMaxDoc(LeafReader in, int maxDoc) {
+ super(in);
+ if (in.getLiveDocs() != null) {
+ // a shorter live-docs Bits would be read past its end for the padding docs; the sidecar
+ // never deletes, and AuxIndexJoinMergePolicy keeps segments with deletions out of these
+ // merges, so this is unreachable rather than merely unlikely
+ throw new IllegalStateException("cannot doc-align a sidecar segment carrying deletions");
+ }
+ this.maxDoc = maxDoc;
+ }
+
+ @Override
+ public int maxDoc() {
+ return maxDoc;
+ }
+
+ @Override
+ public int numDocs() {
+ return maxDoc;
+ }
+
+ @Override
+ public Bits getLiveDocs() {
+ return null;
+ }
+
+ @Override
+ public StoredFields storedFields() {
+ return NO_STORED_FIELDS;
+ }
+
+ @Override
+ public TermVectors termVectors() {
+ return TermVectors.EMPTY;
+ }
+
+ @Override
+ protected void doClose() {
+ // the wrapped reader belongs to the merge, closing it here would pull it from under
+ // IndexWriter; the union only ever decrefs these views anyway
+ }
+
+ @Override
+ public CacheHelper getCoreCacheHelper() {
+ return null; // maxDoc and the field set differ from the wrapped reader's
+ }
+
+ @Override
+ public CacheHelper getReaderCacheHelper() {
+ return null;
+ }
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/ForeignKeyColumn.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/ForeignKeyColumn.java
new file mode 100644
index 00000000000..673749765f1
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/ForeignKeyColumn.java
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.util.BitUtil;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefHash;
+import org.apache.lucene.util.packed.PackedInts;
+import org.apache.lucene.util.packed.PackedLongValues;
+import org.apache.lucene.util.packed.PagedMutable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * From-side state of one (from-segment, to-segment) pair: the from doc values, its live-docs mask,
+ * and the hashed from-side term dictionary. The hash maps each from-side term to its from-ord via
+ * {@link #getFromTermOrdOrDashOne}, so the to-side stage of {@link
+ * JoinIndexUtils#computeDocMapping} can resolve a to-side term to the from-ord it shares. {@link
+ * #fromOrds()} walks each live from-doc's from-ord, which that stage reads to resolve from-ords to
+ * to-docs, into a column of its own.
+ *
+ * <p>One of these is loaded per from-segment and shared by every to-segment paired with it, but
+ * that is only within a query: a pair that never reaches the join index has its from side loaded
+ * again on the next query, and a production run was doing about eighty of these loads a minute. So
+ * both maps here are paged the way {@link JoinIndexUtils.JoinColumnModel} is -- a from-segment of
+ * five million docs held the from-ord map alone as a 20MB {@code int[]}, which G1 can only place as
+ * a run of humongous regions, and eighty of those a minute is what a 2GB heap died of. Packing each
+ * map at the bits its values need takes the same map to about 14MB, in pages no bigger than 128KB.
+ *
+ * @lucene.experimental
+ */
+final class ForeignKeyColumn {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private final BytesRefHash fromTermsHash;
+
+ /** The from-ord of the term at each of {@link #fromTermsHash}'s hash ords. */
+ private final PagedMutable fromOrdByHashOrd;
+
+ /**
+ * {@code fromOrd + 1} at each from-doc, {@code 0} where the doc is deleted or has no value,
+ * running to the last from-doc that has one and no further.
+ */
+ private final PackedLongValues fromOrdPlusOneByFromDoc;
+
+ private final int fromSideMaxDocs;
+ private final int fromValuesCount;
+
+ public ForeignKeyColumn(LeafReaderContext fromContext, String fromField) throws IOException {
+ long startNanos = System.nanoTime();
+ SortedSetDocValues fromDV = DocValues.getSortedSet(fromContext.reader(), fromField);
+ Bits fromLiveDocs = fromContext.reader().getLiveDocs();
+ // dead code, kept until M:N support settles: the reverse (to-side sized) ord map was filled
+ // but never read
+ // long[] fromOrdByToOrd = new long[Math.toIntExact(toDV.getValueCount())];
+ // Arrays.fill(fromOrdByToOrd, -1L);
+ TermsEnum fromTerms = fromDV.termsEnum();
+ int fkVals = Math.toIntExact(fromDV.getValueCount());
+ // BytesRefHash only rehashes once it is exactly half full (count == hashHalfSize), so a table
+ // seeded at size 1 (fkVals == 1) never rehashes and ends up full; a probe for an absent term
+ // then never finds an empty slot and loops forever. Seeding at least 2 lets that rehash kick
+ // in.
+ int capacity =
+ Math.max(
+ 2,
+ BitUtil.nextHighestPowerOfTwo(
+ fkVals * 2 + 2 // don't want to rehash
+ ));
+ // pool,
+ BytesRefHash fromTermsHash =
+ new BytesRefHash(
+ new ByteBlockPool(new ByteBlockPool.DirectAllocator()),
+ capacity,
+ new BytesRefHash.DirectBytesStartArray(capacity));
+ // no sentinel needed: this is only ever read at a hash ord the hash itself handed back, i.e.
+ // one that was written here, so the zero a fresh page starts at is never mistaken for a value
+ PagedMutable fromOrdByHashOrd =
+ new PagedMutable(
+ capacity,
+ JoinIndexUtils.PAGE_SIZE,
+ PackedInts.bitsRequired(fkVals),
+ PackedInts.COMPACT);
+ for (BytesRef term = fromTerms.next(); term != null; term = fromTerms.next()) {
+ fromOrdByHashOrd.set(fromTermsHash.add(term), fromTerms.ord());
+ }
+ // from-docs arrive ascending, so the map is appended rather than addressed: each doc's
+ // from-ord is offset by one so that the zero a skipped doc leaves behind reads back as "no
+ // value", and the tail past the last valued doc is never laid down at all
+ PackedLongValues.Builder fromOrdPlusOneByFromDoc =
+ PackedLongValues.packedBuilder(JoinIndexUtils.PAGE_SIZE, PackedInts.COMPACT);
+ int laidDown = 0;
+ for (int fromDoc = fromDV.nextDoc();
+ fromDoc != DocIdSetIterator.NO_MORE_DOCS;
+ fromDoc = fromDV.nextDoc()) {
+ if (fromLiveDocs != null && !fromLiveDocs.get(fromDoc)) {
+ continue;
+ }
+ while (laidDown < fromDoc) {
+ fromOrdPlusOneByFromDoc.add(0L);
+ laidDown++;
+ }
+ fromOrdPlusOneByFromDoc.add(fromDV.nextOrd() + 1);
+ laidDown++;
+ }
+
+ this.fromTermsHash = fromTermsHash;
+ this.fromOrdByHashOrd = fromOrdByHashOrd;
+ this.fromOrdPlusOneByFromDoc = fromOrdPlusOneByFromDoc.build();
+ this.fromSideMaxDocs = fromContext.reader().maxDoc();
+ this.fromValuesCount = Math.toIntExact(fromDV.getValueCount());
+ if (JoinIndexUtils.diagnosticsEnabled(log)) {
+ // this constructor is the heavy from-side work (hashes the whole term dictionary), so every
+ // line here is one profiler-visible FK load; a segment recurring across queries means its
+ // pairs never get persisted and the load is being repeated in vain
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=fkload fromSeg={} field={} ord={} maxDoc={} values={} pagedBytes={}"
+ + " tookUs={}",
+ JoinIndexUtils.segmentName(fromContext),
+ fromField,
+ fromContext.ord,
+ fromContext.reader().maxDoc(),
+ fromValuesCount,
+ pagedBytesUsed(),
+ (System.nanoTime() - startNanos) / 1_000L);
+ }
+ }
+
+ public int getFromTermOrdOrDashOne(BytesRef value) {
+ int hashOrd = this.fromTermsHash.find(value);
+ if (hashOrd != -1) {
+ return (int) this.fromOrdByHashOrd.get(hashOrd);
+ } else {
+ return -1;
+ }
+ }
+
+ /**
+ * A fresh walk over this from-segment's from-docs, from doc 0. One instance of this class serves
+ * every to-segment paired with this from-segment, and {@link JoinIndexUtils#computeDocMapping}
+ * walks it twice per pair -- once to measure, once to lay out -- so the map is read through
+ * cursors rather than copied: it used to be cloned per pair and rewritten in place, which cost a
+ * full {@code int[maxDoc]} per pair (20MB on a five-million-doc from-segment) regardless of how
+ * little the pair matched.
+ */
+ public FromOrds fromOrds() {
+ return new FromOrds(fromOrdPlusOneByFromDoc);
+ }
+
+ /**
+ * A forward-only walk over from-docs and the from-ord each carries, in doc order from doc 0. The
+ * walk ends at the last from-doc with a value: nothing past it has one, and neither of {@link
+ * JoinIndexUtils#computeDocMapping}'s passes has anything to do at a doc without one.
+ */
+ static final class FromOrds {
+ private final PackedLongValues.Iterator fromOrdsPlusOne;
+ private int fromDoc = -1;
+ private int fromOrd = -1;
+
+ FromOrds(PackedLongValues fromOrdPlusOneByFromDoc) {
+ this.fromOrdsPlusOne = fromOrdPlusOneByFromDoc.iterator();
+ }
+
+ /** Advances to the next from-doc, or returns false once the walk is spent. */
+ boolean next() {
+ if (!fromOrdsPlusOne.hasNext()) {
+ return false;
+ }
+ fromDoc++;
+ fromOrd = (int) fromOrdsPlusOne.next() - 1;
+ return true;
+ }
+
+ /** The from-doc the walk stands on. */
+ int fromDoc() {
+ return fromDoc;
+ }
+
+ /** Its from-ord, or {@code -1} where the doc is deleted or carries no value. */
+ int fromOrd() {
+ return fromOrd;
+ }
+ }
+
+ public int fromSideMaxDocs() {
+ return fromSideMaxDocs;
+ }
+
+ /**
+ * What the two paged maps cost on the heap. Excludes the term hash, whose arrays are {@link
+ * BytesRefHash}'s own and still flat.
+ */
+ public long pagedBytesUsed() {
+ return fromOrdByHashOrd.ramBytesUsed() + fromOrdPlusOneByFromDoc.ramBytesUsed();
+ }
+
+ public int getFromValuesCount() {
+ return fromValuesCount;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/FromLeafJoinContext.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/FromLeafJoinContext.java
new file mode 100644
index 00000000000..0d65f7e9fc9
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/FromLeafJoinContext.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.lang.invoke.MethodHandles;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.search.Weight;
+import org.jspecify.annotations.NonNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * @lucene.experimental
+ */
+class FromLeafJoinContext {
+ final JoinIndexUtils.CacheAndCount matches;
+ final ForeignKeyColumn fkColumn;
+ final int ord;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public FromLeafJoinContext(
+ JoinIndexUtils.CacheAndCount matches, ForeignKeyColumn fkColumn, int ord) {
+ this.matches = matches;
+ this.fkColumn = fkColumn;
+ this.ord = ord;
+ }
+
+ static @NonNull FromLeafJoinContext heavyLoadFromLeaf(
+ Weight fromWeight, String fromField, LeafReaderContext ctx, boolean loadFkColumn) {
+ try {
+ JoinIndexUtils.CacheAndCount docset = JoinIndexUtils.computeDocIdSet(fromWeight, ctx);
+ boolean loadFk = docset != null && docset.count() > 0 && loadFkColumn;
+ if (JoinIndexUtils.diagnosticsEnabled(log)) {
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=fromLeaf fromSeg={} ord={} fromMatches={} missingPair={}"
+ + " fkLoaded={}",
+ JoinIndexUtils.segmentName(ctx),
+ ctx.ord,
+ docset == null ? -1 : docset.count(),
+ loadFkColumn,
+ loadFk);
+ }
+ if (loadFk) {
+ // waste case: noone to-seg read FK,
+ return new FromLeafJoinContext(docset, new ForeignKeyColumn(ctx, fromField), ctx.ord);
+ } else {
+ return new FromLeafJoinContext(docset, null, ctx.ord);
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumWriter.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumWriter.java
new file mode 100644
index 00000000000..87efd91f5c5
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumWriter.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+
+/**
+ * Isolates every write-side representation of a join batch behind this one write-only surface, so
+ * {@link AuxIndexManager} and {@link JoinIndexUtils} deal only in {@link JoinColumnModel} and never
+ * need to import a particular field-writing API themselves.
+ *
+ * @lucene.experimental
+ */
+abstract class JoinColumWriter {
+
+ protected JoinColumWriter() {}
+
+ /**
+ * Builds the pair columns for every entry in {@code mappings} (doc-map plus edges, keyed by pair
+ * field name) and writes them as a single indexing call + flush, so a batch's columns all land at
+ * doc 0 of their own sidecar segment.
+ */
+ abstract void writeJoinColumns(IndexWriter writer, Map<String, JoinColumnModel> mappings)
+ throws IOException;
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnDocWriter.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnDocWriter.java
new file mode 100644
index 00000000000..4e8290695ef
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnDocWriter.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.SortedNumericDocValuesField;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.DocEdges;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+
+/**
+ * Sibling of {@code AIJoinColumnWriter} writing the same pair columns through the plain {@link
+ * Document} / {@link IndexWriter#addDocuments} API instead of {@code
+ * org.apache.lucene.document.column}. The batch is handed over as a single {@code addDocuments}
+ * call: per its block semantics it is indexed atomically, with no flush allowed to land in the
+ * middle of it, so -- exactly like {@code AIJoinColumnWriter}'s single {@code addBatch} -- the
+ * whole batch is guaranteed to end up doc-for-doc (batch position == doc id) in one sidecar
+ * segment, keeping doc 0's edges and every from-doc id aligned the same way.
+ *
+ * <p><b>The batch is streamed, not materialized.</b> It spans the whole from-segment -- one
+ * document per from-side doc id, since that id <em>is</em> the address a pair column is read at --
+ * so building it as a {@code List<Document>} cost a {@link Document} (and its own field list) per
+ * from-doc: on a 5M-doc from-segment, some 400MB of scaffolding, live for the entire {@code
+ * addDocuments} call, to write a segment that lands on disk at about 1.3MB. That is what exhausted
+ * a 2GB heap under load. {@link BatchDocuments} mints the documents one at a time instead, reusing
+ * one {@link Document} and one field per pair column, so what the batch costs the heap no longer
+ * depends on how long the from-segment is -- the sparse docvalues buffer the indexing chain fills
+ * as it consumes them does, but that is a fraction of the same size.
+ *
+ * @lucene.experimental
+ */
+final class JoinColumnDocWriter extends JoinColumWriter {
+
+ JoinColumnDocWriter() {}
+
+ @Override
+ void writeJoinColumns(IndexWriter writer, Map<String, JoinColumnModel> mappings)
+ throws IOException {
+ int batchNumDocs = 0;
+ for (Map.Entry<String, JoinColumnModel> entry : mappings.entrySet()) {
+ batchNumDocs = Math.max(batchNumDocs, entry.getValue().maxDoc());
+ }
+ assert mappings.isEmpty() || batchNumDocs > 0
+ : "a batch with columns to write needs a doc 0 to carry their edges: " + mappings.keySet();
+ // a single block: IndexWriter guarantees no intermediate flush splits it across segments
+ writer.addDocuments(new BatchDocuments(mappings, batchNumDocs));
+ // seal the batch into a segment of its own, so the next batch starts again at doc 0 -- the
+ // invariant every column depends on, since a sidecar doc number IS a from-doc id. This used
+ // to be a commit(), which sealed the segment only incidentally and paid an fsync of every new
+ // file plus a segments_N for it, on the query thread, inside AuxIndexManager's write lock.
+ // flush() writes the segment and nothing else; durability and file reclamation are the
+ // periodic commit's job.
+ writer.flush();
+ }
+
+ /**
+ * The batch as a lazy {@link Iterable}: {@code batchNumDocs} documents, each carrying whatever
+ * the pair columns hold at that from-doc id, generated as {@link IndexWriter} asks for them.
+ *
+ * <p>Every document is the same {@link Document} instance, cleared and refilled. That is safe
+ * because {@code addDocuments} consumes each document fully -- the indexing chain reads its
+ * fields and buffers their values -- before pulling the next one, and it keeps no reference
+ * afterwards. {@link #iterator()} builds its own cursors, so the batch can be iterated again if
+ * {@code IndexWriter} ever needs to.
+ */
+ private static final class BatchDocuments implements Iterable<Document> {
+
+ private final Map<String, JoinColumnModel> mappings;
+ private final int numDocs;
+
+ BatchDocuments(Map<String, JoinColumnModel> mappings, int numDocs) {
+ this.mappings = mappings;
+ this.numDocs = numDocs;
+ }
+
+ @Override
+ public Iterator<Document> iterator() {
+ List<PairColumn> columns = new ArrayList<>(mappings.size());
+ for (Map.Entry<String, JoinColumnModel> entry : mappings.entrySet()) {
+ columns.add(new PairColumn(entry.getKey(), entry.getValue()));
+ }
+ return new Iterator<>() {
+
+ private final Document doc = new Document();
+ private int nextDocNum = 0;
+
+ @Override
+ public boolean hasNext() {
+ return nextDocNum < numDocs;
+ }
+
+ @Override
+ public Document next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ int docNum = nextDocNum++;
+ doc.clear();
+ for (PairColumn column : columns) {
+ column.addTo(doc, docNum);
+ }
+ return doc;
+ }
+ };
+ }
+ }
+
+ /**
+ * One pair's contribution to the batch: the doc-map column walked in step with the batch, and the
+ * edges companions that go to doc 0 -- written even when the pair maps nothing, so a once-built
+ * pair is detectable in the join index and never rebuilt.
+ */
+ // TODO don't write minusones columns for tombstones!!
+ private static final class PairColumn {
+
+ private final String pairFieldName;
+ private final DocEdges edges;
+ private final SortedNumericDocValues toDocByFromDoc;
+
+ /**
+ * Reused across the batch: a from-doc carries at most one to-doc (the mapping is single-valued
+ * until M:N pairs are supported), so no document ever needs two of these at once.
+ */
+ private final SortedNumericDocValuesField toDoc;
+
+ /** The next from-doc this column has a value at, or {@code NO_MORE_DOCS} when spent. */
+ private int atFromDoc;
+
+ PairColumn(String pairFieldName, JoinColumnModel mapping) {
+ this.pairFieldName = pairFieldName;
+ this.edges = mapping.edges();
+ this.toDocByFromDoc = mapping.toDocByFromDoc();
+ this.toDoc =
+ new SortedNumericDocValuesField(
+ JoinIndexUtils.TO_DOC_VAL_BY_FROM_DOCNUM + pairFieldName, 0L);
+ this.atFromDoc = nextFromDoc();
+ }
+
+ /** Adds this column's fields for one batch position, if it has any there. */
+ void addTo(Document doc, int docNum) {
+ if (docNum == 0) {
+ addEdges(doc, JoinIndexUtils.FROM_EDGES_PREFIX + pairFieldName, edges.fromDocEdges());
+ addEdges(doc, JoinIndexUtils.TO_EDGES_PREFIX + pairFieldName, edges.toDocEdges());
+ addEdges(doc, JoinIndexUtils.TO_COUNT_PREFIX + pairFieldName, new int[] {edges.toCount()});
+ }
+ if (atFromDoc == docNum) {
+ assert toDocByFromDoc.docValueCount() == 1
+ : "the reused field carries one to-doc per from-doc, got "
+ + toDocByFromDoc.docValueCount();
+ toDoc.setLongValue(nextValue());
+ doc.add(toDoc);
+ atFromDoc = nextFromDoc();
+ }
+ }
+
+ /**
+ * Adds a pair's {min, max} (or count) values to doc 0, mirroring {@code AIJoinColumnWriter}'s
+ * {@code edgesColumn}, which puts both values at doc 0 too. Not reusable the way {@link #toDoc}
+ * is: doc 0 holds both edges of a pair under the same field name, so they must be two fields.
+ */
+ private static void addEdges(Document doc, String fieldName, int[] values) {
+ for (int value : values) {
+ doc.add(new SortedNumericDocValuesField(fieldName, value));
+ }
+ }
+
+ // the mapping is an in-memory array (JoinIndexUtils.JoinColumnModel), so neither of these can
+ // actually fail; they only inherit IOException from the docvalues read API
+ private int nextFromDoc() {
+ try {
+ return toDocByFromDoc.nextDoc();
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private long nextValue() {
+ try {
+ return toDocByFromDoc.nextValue();
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnIndexer.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnIndexer.java
new file mode 100644
index 00000000000..e21198f171f
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinColumnIndexer.java
@@ -0,0 +1,411 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.JoinSegmentReference;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.SegmentsTuple;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builds missing pair columns for an {@link AuxIndexManager}: dedups concurrent builders per pair
+ * field name, computes the in-memory {@link JoinColumnModel}s, double-checks a fresh join-index
+ * searcher for pairs persisted since the caller looked, and persists what is still missing through
+ * {@link AuxIndexManager#writeBatch}.
+ *
+ * @lucene.experimental
+ */
+final class JoinColumnIndexer {
+
+ private final AuxIndexManager joinIndex;
+
+ /**
+ * Dedups concurrent builders per pair field name: the thread that installs the future writes the
+ * pair, others wait on it. A future completes as soon as the pair's in-memory {@link
+ * JoinColumnModel} is ready -- before the owning thread persists and refreshes -- so completion
+ * does not imply the column is visible through the index's searcher manager yet. Entries are
+ * transient: the owner removes its claims once the pair is persisted and refreshed (and on build
+ * failure), so the map only ever holds in-flight builds and never pins models in heap. A builder
+ * that races a not-yet-visible refresh and re-claims an already-persisted pair is stopped from
+ * writing a duplicate column by {@code #writeJoinSegments}'s fresh-searcher double-check, not by
+ * this map.
+ */
+ private final ConcurrentHashMap<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>>
+ pairBuilds = new ConcurrentHashMap<>();
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ JoinColumnIndexer(AuxIndexManager joinIndex) {
+ this.joinIndex = joinIndex;
+ }
+
+ /**
+ * How many of the given pair field names have a build currently in flight in {@link #pairBuilds}
+ * (claims are removed once persisted). Diagnostic only: a pair counted here but still reported
+ * missing by {@link JoinIndexUtils#extractExistingJoinColumns} means the caller is about to redo
+ * from-side work for a pair some other thread is building right now.
+ */
+ int countClaimedBuilds(Set<String> pairFieldNames) {
+ int claimed = 0;
+ for (String pairFieldName : pairFieldNames) {
+ if (pairBuilds.containsKey(pairFieldName)) {
+ claimed++;
+ }
+ }
+ return claimed;
+ }
+
+ /**
+ * Builds and persists the given missing pair columns, keyed by pair field name to their
+ * (from-segment, to-segment) leaf ordinals. Pairs concurrently built by another thread are
+ * awaited, not rebuilt; awaiting yields the builder's in-memory model as soon as it is computed,
+ * possibly before the builder has committed it, so on return pairs built by this thread are
+ * refreshed into the internal searcher manager, while awaited ones may not be visible there yet.
+ *
+ * <p>Claims in {@link #pairBuilds} are transient -- the owner removes them once its pairs are
+ * persisted and refreshed, keeping the map bounded by in-flight builds. That makes re-claiming an
+ * already-persisted pair an expected event: the caller decided "missing" against {@code
+ * observedAbsentSearcher}, which may predate another thread's completed build. So after claiming,
+ * this method re-checks a fresh searcher and skips persisting any pair that turns out already
+ * written. Since only the claim owner ever writes a pair, absence in a post-claim fresh searcher
+ * is conclusive, so no duplicate pair column can be written; rediscovered pairs still cost their
+ * in-memory recompute, which the caller gets back in the result like any built pair. As a
+ * shortcut, when {@link AuxIndexManager#acquire()} returns {@code observedAbsentSearcher} itself,
+ * nothing was committed since that searcher was current, and the re-check scan is skipped.
+ *
+ * @param observedAbsentSearcher the join-index searcher in which the caller established that
+ * {@code missingPairs} are absent; possibly stale by now. Pass {@code null} when absence
+ * wasn't verified against a live searcher -- that forces the re-check scan.
+ * @return in memory data for just written segemts
+ */
+ Map<String, JoinColumnModel> buildAndPersistJoinColumns(
+ Map<String, SegmentsTuple> missingPairs,
+ IndexReader fromReader,
+ IndexReader toReader,
+ String toField,
+ String traceCtxId,
+ IndexSearcher observedAbsentSearcher,
+ Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures)
+ throws IOException, ExecutionException, InterruptedException {
+ long startNanos = System.nanoTime();
+ Claims claims = claimPairs(missingPairs.keySet());
+ Map<String, JoinColumnModel> loadedMappings = Map.of();
+ Set<String> writtenPairs = Set.of();
+ // kept apart because they answer different questions: computing a model walks the from
+ // segment once per pair, while persisting it walks the whole from segment again as sidecar
+ // documents, whatever the pair matched. Which of the two dominates decides where the next
+ // round of work goes.
+ long computeNanos = 0;
+ long persistNanos = 0;
+ try {
+ if (!claims.owned().isEmpty()) {
+ long computeStartNanos = System.nanoTime();
+ loadedMappings =
+ computeOwnedModels(
+ claims.owned().keySet(),
+ missingPairs,
+ fromReader,
+ toReader,
+ toField,
+ fromColumnFutures);
+ computeNanos = System.nanoTime() - computeStartNanos;
+ completeClaimFutures(claims.owned(), loadedMappings);
+ Map<String, JoinColumnModel> unwrittenMappings =
+ dropAlreadyPersisted(loadedMappings, claims.owned().keySet(), observedAbsentSearcher);
+ long persistStartNanos = System.nanoTime();
+ persistBatch(unwrittenMappings);
+ persistNanos = System.nanoTime() - persistStartNanos;
+ writtenPairs = unwrittenMappings.keySet();
+ releaseClaims(claims.owned());
+ }
+ } catch (Throwable t) {
+ withdrawClaims(claims.owned(), t);
+ throw t;
+ }
+ long builtNanos = System.nanoTime() - startNanos;
+ Map<String, JoinColumnModel> result = new LinkedHashMap<>(loadedMappings);
+ collectAwaited(claims.awaited(), result);
+ logBuildDiagnostics(
+ traceCtxId,
+ missingPairs.size(),
+ loadedMappings,
+ claims.awaited().size(),
+ writtenPairs,
+ startNanos,
+ builtNanos,
+ computeNanos,
+ persistNanos);
+ return result;
+ }
+
+ /**
+ * The claim/await split of one build request: {@code owned} holds the futures this thread
+ * installed into {@link #pairBuilds} (it must build, persist, and eventually remove them), {@code
+ * awaited} the futures of pairs some other thread is already building.
+ */
+ private record Claims(
+ Map<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> owned,
+ List<CompletableFuture<Map.Entry<String, JoinColumnModel>>> awaited) {}
+
+ /**
+ * Races a claim into {@link #pairBuilds} for every requested pair: the thread whose future lands
+ * owns that pair's build, a pair already claimed by another thread is awaited instead.
+ */
+ private Claims claimPairs(Set<String> pairFieldNames) {
+ Map<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> owned =
+ new LinkedHashMap<>();
+ List<CompletableFuture<Map.Entry<String, JoinColumnModel>>> awaited = new ArrayList<>();
+ for (String pairFieldName : pairFieldNames) {
+ CompletableFuture<Map.Entry<String, JoinColumnModel>> created = new CompletableFuture<>();
+ CompletableFuture<Map.Entry<String, JoinColumnModel>> existing =
+ pairBuilds.putIfAbsent(pairFieldName, created);
+ if (existing == null) {
+ owned.put(pairFieldName, created);
+ } else {
+ awaited.add(existing);
+ }
+ }
+ return new Claims(owned, awaited);
+ }
+
+ /** Computes the in-memory {@link JoinColumnModel} for every owned pair. */
+ private static Map<String, JoinColumnModel> computeOwnedModels(
+ Set<String> ownedPairs,
+ Map<String, SegmentsTuple> missingPairs,
+ IndexReader fromReader,
+ IndexReader toReader,
+ String toField,
+ Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures)
+ throws IOException, ExecutionException, InterruptedException {
+ Map<String, JoinColumnModel> loadedMappings = new LinkedHashMap<>();
+ JoinIndexUtils.ToDocInvertor toInvertor = new JoinIndexUtils.ToDocInvertor(toField);
+ for (String pairFieldName : ownedPairs) {
+ SegmentsTuple position = missingPairs.get(pairFieldName);
+ LeafReaderContext toContext = toReader.leaves().get(position.toLeafOrd());
+ LeafReaderContext fromContext = fromReader.leaves().get(position.fromLeafOrd());
+ Future<FromLeafJoinContext> fromLeafJoinContextFuture =
+ fromColumnFutures.get(fromContext.ord);
+ assert fromLeafJoinContextFuture != null;
+ FromLeafJoinContext fkCtx = fromLeafJoinContextFuture.get();
+ JoinIndexUtils.JoinColumnModel mapping =
+ JoinIndexUtils.computeDocMapping(toContext, toField, fkCtx.fkColumn, toInvertor);
+ loadedMappings.put(pairFieldName, mapping);
+ }
+ return loadedMappings;
+ }
+
+ /**
+ * Completes every owned claim future with its computed model. Runs before {@link #persistBatch}:
+ * waiters only need the in-memory model, so they don't pay for the owner's commit + refresh;
+ * completion does NOT imply the column is visible through the searcher manager yet.
+ */
+ private static void completeClaimFutures(
+ Map<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> owned,
+ Map<String, JoinColumnModel> loadedMappings) {
+ for (Map.Entry<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> entry :
+ owned.entrySet()) {
+ entry
+ .getValue()
+ .complete(
+ new AbstractMap.SimpleImmutableEntry<>(
+ entry.getKey(), loadedMappings.get(entry.getKey())));
+ }
+ }
+
+ /**
+ * Double-checks the owned pairs against a fresh searcher, returning only the mappings still
+ * unwritten: claims are dropped once persisted, so a caller deciding "missing" on a stale
+ * searcher can re-claim an already-persisted pair; owning the claim means no one else can write
+ * it concurrently, so what the fresh searcher lacks is conclusively unwritten.
+ *
+ * <p>It's a race condition tradeoff:
+ *
+ * <ol>
+ * <li>this thread didn't see a column and decided to compete for it.
+ * <li>another thread might have persisted it in the meantime, and removed it's claim from
+ * pairBuilds.
+ * <li>thread 1. calculate a join column model, and ready to write it
+ * <li>if a column occur in already persisted: it skip wtiting it, but returns the calculated
+ * memory model to the caller
+ * </ol>
+ *
+ * overall, under high concurrency we waste some computation for sweeping pairBuilds;
+ * alternatively we need to invent a way to remove entries from it somewhere later without a race
+ */
+ private Map<String, JoinColumnModel> dropAlreadyPersisted(
+ Map<String, JoinColumnModel> loadedMappings,
+ Set<String> ownedPairs,
+ IndexSearcher observedAbsentSearcher)
+ throws IOException {
+ Map<String, JoinColumnModel> unwrittenMappings = loadedMappings;
+ IndexSearcher freshJoinSearcher = joinIndex.acquire();
+ try {
+ if (!Objects.equals(freshJoinSearcher, observedAbsentSearcher)) {
+ Map<String, JoinSegmentReference> alreadyPersisted =
+ JoinIndexUtils.extractExistingJoinColumns(freshJoinSearcher, ownedPairs::contains);
+ if (!alreadyPersisted.isEmpty()) {
+ unwrittenMappings = new LinkedHashMap<>(loadedMappings);
+ unwrittenMappings.keySet().removeAll(alreadyPersisted.keySet());
+ }
+ } // else: same searcher instance the caller saw the pairs absent in -- nothing was
+ // committed since, so the pairs are provably unwritten and the scan is redundant
+ } finally {
+ joinIndex.release(freshJoinSearcher);
+ }
+ return unwrittenMappings;
+ }
+
+ /**
+ * Persists the still-unwritten mappings, if any. All pairs of a batch go in together: pair
+ * columns are addressed by from-side doc id, so a batch must start at doc 0 of its sidecar
+ * segment, which {@link AuxIndexManager#writeBatch} guarantees by flushing one batch per commit.
+ */
+ // TODO it can be more asynchronous, besides of pairBuilds removal - and it's a problem
+ // would be great if many concurrent threads merge lists and write it at once,
+ // since it's synchronized bottleneck for now.
+ // and the thread don't even need to wait until writeBatch() is done (beside of pairBuilds
+ // removal, you know)
+ // TODO flush every single field to get single field segments
+ private void persistBatch(Map<String, JoinColumnModel> unwrittenMappings) throws IOException {
+ if (!unwrittenMappings.isEmpty()) {
+ joinIndex.writeBatch(unwrittenMappings);
+ }
+ }
+
+ /**
+ * Drops the owned claims after a successful build: every owned pair is now persisted (just
+ * written, or found persisted by the double-check) and visible to any searcher acquired from here
+ * on, and duplicate writes are prevented by the double-check, not by retained entries -- dropping
+ * them keeps {@link #pairBuilds} bounded by in-flight builds and unpins the models from heap.
+ * Waiters already holding the futures are unaffected.
+ */
+ private void releaseClaims(
+ Map<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> owned) {
+ for (Map.Entry<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> entry :
+ owned.entrySet()) {
+ pairBuilds.remove(entry.getKey(), entry.getValue());
+ }
+ }
+
+ /**
+ * Withdraws the owned claims on failure so a later query can retry the build; {@code
+ * completeExceptionally} only takes effect when the failure happened before {@link
+ * #completeClaimFutures} -- if writeBatch failed, waiters have already observed success with a
+ * valid in-memory model, and only the withdrawn claim (and the caller's rethrow) records that the
+ * column was never persisted.
+ */
+ private void withdrawClaims(
+ Map<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> owned, Throwable t) {
+ for (Map.Entry<String, CompletableFuture<Map.Entry<String, JoinColumnModel>>> entry :
+ owned.entrySet()) {
+ pairBuilds.remove(entry.getKey(), entry.getValue());
+ entry.getValue().completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Joins the pairs claimed by other threads and merges their models into {@code result},
+ * unwrapping a builder's failure into this thread's checked contract.
+ */
+ private static void collectAwaited(
+ List<CompletableFuture<Map.Entry<String, JoinColumnModel>>> awaited,
+ Map<String, JoinColumnModel> result)
+ throws IOException {
+ for (CompletableFuture<Map.Entry<String, JoinColumnModel>> future : awaited) {
+ try {
+ Map.Entry<String, JoinColumnModel> entry = future.join();
+ result.put(entry.getKey(), entry.getValue());
+ } catch (CompletionException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof IOException ioe) {
+ throw ioe;
+ }
+ if (cause instanceof RuntimeException re) {
+ throw re;
+ }
+ throw new IOException(cause);
+ }
+ }
+ }
+
+ /**
+ * Emits the {@code evt=build} diagnostic line. The built/awaited split matters: an awaited pair
+ * cost this thread only the wait, so folding the two together would attribute another thread's
+ * build work to this query; pairsBuilt counts models computed by this thread, writtenPairs what
+ * actually reached the sidecar -- the difference is pairs the double-check found already
+ * persisted.
+ */
+ private static void logBuildDiagnostics(
+ String traceCtxId,
+ int pairsRequested,
+ Map<String, JoinColumnModel> loadedMappings,
+ int pairsAwaited,
+ Set<String> writtenPairs,
+ long startNanos,
+ long builtNanos,
+ long computeNanos,
+ long persistNanos) {
+ if (!JoinIndexUtils.diagnosticsEnabled(log) || pairsRequested == 0) {
+ return;
+ }
+ long toCount = 0;
+ long modelBytes = 0;
+ int sparseModels = 0;
+ for (JoinColumnModel model : loadedMappings.values()) {
+ toCount += model.edges().toCount();
+ modelBytes += model.ramBytesUsed();
+ if (model.isSparse()) {
+ sparseModels++;
+ }
+ }
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=build ctx={} pairsRequested={} pairsBuilt={} pairsAwaited={}"
+ + " builtMs={} computeMs={} persistMs={} awaitedMs={} toCount={} sparseModels={}"
+ + " modelBytes={} writtenPairs={}",
+ traceCtxId == null ? "-" : traceCtxId,
+ pairsRequested,
+ loadedMappings.size(),
+ pairsAwaited,
+ builtNanos / 1_000_000L,
+ computeNanos / 1_000_000L,
+ persistNanos / 1_000_000L,
+ (System.nanoTime() - startNanos - builtNanos) / 1_000_000L,
+ toCount,
+ sparseModels,
+ modelBytes,
+ writtenPairs);
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexScorerSupplier.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexScorerSupplier.java
new file mode 100644
index 00000000000..cb727bb36a8
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexScorerSupplier.java
@@ -0,0 +1,941 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.AbstractMap.SimpleEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.search.ConstantScoreScorer;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreMode;
+import org.apache.lucene.search.Scorer;
+import org.apache.lucene.search.ScorerSupplier;
+import org.apache.lucene.search.TwoPhaseIterator;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.JoinSegmentReference;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.SegmentsTuple;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.DocEdges;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Joins all from-side segments to the single to-side segment.
+ *
+ * <h2>Instrumentation</h2>
+ *
+ * Emits TRACE lines prefixed {@code AUXIJOIN} in logfmt (space-separated {@code key=value}), one
+ * {@code evt=} per line, so a run can be parsed without a logging-config change:
+ *
+ * <ul>
+ * <li>{@code evt=build} -- a join-index write happened; carries the wall time it cost. Emitted by
+ * {@code AuxIndexManager#writeJoinSegments}, the chokepoint both build paths share: {@code
+ * cause=eager-create-weight} for the bulk build {nolink AuxIndexManager#ensureJoinSegments}
+ * does at {@link AuxIndexJoinQuery#createWeight} time, and {@code cause=lazy-to-segment} for
+ * the per-context fallback below. In a steady run the eager path does all the work and the
+ * lazy one never fires, so a log with no {@code cause=lazy-to-segment} line is the expected
+ * shape.
+ * <li>{@code evt=ctx} -- this context finished setting up: how many (from, to) pairs contributed,
+ * how many the a-priori from-edge check dropped before any column was opened, and how loose
+ * the resulting approximation is.
+ * <li>{@code evt=drain} -- one join column was read through during confirmation, and whether that
+ * read confirmed the doc under test (an early exit) or not.
+ * <li>{@code evt=done} -- confirmation reached a terminal state for this context: every column
+ * has been drained, so the half-read union has converged. Carries the per-context totals.
+ * </ul>
+ *
+ * Every line carries {@code ctx=}, the {@link #ctxId} of the context it belongs to ({@code -} on an
+ * eager build, which precedes every context), so a parser can attribute drains exactly instead of
+ * assuming one context per to-segment is alive at a time.
+ *
+ * <p>The counters are best-effort: they are plain fields on a context confined to one to-segment,
+ * and so to one search thread, but nothing enforces that. A context whose confirmation never
+ * converges emits no {@code evt=done} line -- absence of one is itself the signal that laziness
+ * paid off for that segment, at the cost of that context's final counters going unreported.
+ *
+ * @lucene.experimental
+ */
+class JoinIndexScorerSupplier extends ScorerSupplier {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Per-JVM sequence behind {@link #ctxId}. Only has to distinguish contexts alive at the same
+ * time, so wrapping after 2^63 is not a concern; base 36 just keeps the log lines short.
+ */
+ private static final AtomicLong CTX_SEQ = new AtomicLong();
+
+ /**
+ * Identifies this context in the log. Every {@code evt=ctx} / {@code evt=drain} / {@code
+ * evt=done} line carries it, so drains can be attributed to their context exactly, rather than by
+ * assuming only one context per to-segment is alive -- which only holds when a single query runs
+ * at a time.
+ */
+ private final String ctxId = Long.toString(CTX_SEQ.incrementAndGet(), 36);
+
+ final LeafReaderContext toContext;
+ final Query fromQuery;
+ final IndexSearcher fromSearcher;
+ private final Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures;
+ private final ScoreMode scoreMode;
+ private final float boost;
+ private IndexSearcher lastSeenJoinSearcher;
+ final String fromField;
+ final String toField;
+ private final AuxIndexManager joinIndex;
+
+ // TODO all of these might be final since they are set in the constructor
+ private int firstToDoc = DocIdSetIterator.NO_MORE_DOCS;
+ private int lastToDoc = -1;
+ private long matchedToDocsCount = 0;
+ private FixedBitSet falsePositiveToDocsBits = null;
+
+ // ---- instrumentation, best effort; see the class javadoc for the emitted format ----
+ /** nanos spent inside {@code AuxIndexManager#writeJoinSegments} on behalf of this context */
+ private long joinIndexBuildNanos;
+
+ /** pairs that had a from-side match before the a-priori from-edge check ran */
+ private int joinLeafsCreated;
+
+ /** pairs the a-priori from-edge check dropped, i.e. columns never opened at all */
+ private int leafsDroppedApriori;
+
+ /** calls to {@link LazyConfirmationIterator#matches()} */
+ private int confirmCalls;
+
+ /** those answered from the half-read union alone, with no column read */
+ private int confirmFreeHits;
+
+ /** columns read through during confirmation */
+ private int leafsDrained;
+
+ /** from-docs walked while draining, i.e. the column reads laziness is trying to avoid */
+ private long fromDocsWalked;
+
+ /**
+ * cells whose column was reaped from under this query and had to be rebuilt ({@link
+ * LeafJoin#bind}). Steady-state zero: a non-zero count means the reaper is queueing pairs that
+ * are still in use, and every one of them costs a column build on the query path.
+ */
+ private int rebindsAfterReap;
+
+ // how many cells handed their built model back for the column on disk, see adoptOnDiskColumn
+ private int modelsReleasedToDisk;
+
+ /** set once the {@code evt=done} line has been emitted, so it is emitted at most once */
+ private boolean reported;
+
+ /** ordered by from-cost descending */
+ private final List<LeafJoin> leafJoins = new ArrayList<>();
+
+ private final IndexReader toReader;
+
+ private final class LazyConfirmationIterator extends TwoPhaseIterator {
+ FixedBitSet falseNegToDocsBits = null;
+ private int shift;
+
+ private LazyConfirmationIterator(DocIdSetIterator approximation) {
+ super(approximation);
+ }
+
+ @Override
+ public boolean matches() throws IOException {
+ confirmCalls++;
+ if (leafJoins.isEmpty()) {
+ assert falsePositiveToDocsBits.get(approximation.docID());
+ confirmFreeHits++;
+ return true; /// aproximation is a refined true pos already
+ }
+ if (falseNegToDocsBits != null) {
+ if (falseNegToDocsBits.get(approximation.docID() - shift)) {
+ confirmFreeHits++;
+ return true;
+ } // otherwise we don't know if 0 is real false
+ }
+
+ IndexSearcher freshSearcher = JoinIndexScorerSupplier.this.joinIndex.acquire();
+ try {
+ // the result matters here too: a cell whose column the reaper dropped is rebuilt by the
+ // refresh, and until it is bound the cell still addresses the segment the column left
+ TaskRefreshResult refreshed = refreshJoinTasksReferences(freshSearcher);
+ for (Entry<LeafJoin, JoinColumnModel> entry : refreshed.justWritten) {
+ entry.getKey().bind(entry.getValue());
+ }
+ assert Objects.equals(JoinIndexScorerSupplier.this.lastSeenJoinSearcher, freshSearcher);
+ for (LeafJoin joinTask : new ArrayList<>(leafJoins)) {
+ if (falseNegToDocsBits == null) {
+ this.shift = approximation.docID();
+ falseNegToDocsBits = new FixedBitSet(lastToDoc + 1 - shift);
+ }
+ int walked = joinTask.dumpMatchesInto(falseNegToDocsBits, shift, approximation.docID());
+ if (!joinTask.fromSegIterIsNotExausted()) {
+ JoinIndexScorerSupplier.this.dropJoinLeaf(joinTask);
+ leafsDrained++;
+ }
+ fromDocsWalked += walked;
+ if (!leafJoins.isEmpty()) {
+ if (falseNegToDocsBits.get(approximation.docID() - shift)) {
+ logDrain(joinTask, walked, true);
+ return true;
+ } // otherwise we don't know if 0 is real false
+ }
+ logDrain(joinTask, walked, false);
+ }
+ } catch (ExecutionException e) {
+ throw new RuntimeException(e);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ } finally {
+ JoinIndexScorerSupplier.this.joinIndex.release(freshSearcher);
+ }
+ // drop all to masks, got no hit - it means it's a true negative now.
+
+ falsePositiveToDocsBits.clear(shift, lastToDoc + 1);
+ FixedBitSet.orRange(
+ falseNegToDocsBits, 0, falsePositiveToDocsBits, shift, lastToDoc - shift + 1);
+
+ boolean matched = falsePositiveToDocsBits.get(approximation.docID());
+ // every column has now been drained: the half-read union has converged, so from here on
+ // every answer -- true or false -- is a free lookup and there is nothing left to prune
+ logConfirmationDone(matched ? "converged-on-match" : "converged-on-refutation");
+ return matched;
+ }
+
+ /**
+ * Reports one column read. {@code confirmed=true} is the early exit the lazy variant exists
+ * for: the doc under test was found before the remaining {@code cellsLeft} columns were
+ * touched.
+ */
+ private void logDrain(LeafJoin joinTask, int walked, boolean confirmed) {
+ if (!JoinIndexUtils.diagnosticsEnabled(log)) {
+ return;
+ }
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=drain ctx={} toSeg={} pair={} confirmed={} walked={} colToCount={}"
+ + " cellsLeft={} hCard={} confirmCalls={}",
+ ctxId,
+ JoinIndexUtils.segmentName(toContext),
+ joinTask.pairFieldName,
+ confirmed,
+ walked,
+ joinTask.toCount(),
+ leafJoins.size(),
+ falseNegToDocsBits == null ? 0 : falseNegToDocsBits.cardinality(),
+ confirmCalls);
+ }
+
+ @Override
+ public float matchCost() {
+ return matchedToDocsCount;
+ }
+ }
+
+ /**
+ * Represents a cell in the join matrix bounded to from and to segments. Resolved exactly once,
+ * either from the join index ({@link #resolveFromReadingJoinIndex}: just the pair's {@link
+ * DocEdges}, with real docvalues opened later, on demand, through {@link #joinSegmentRef} --
+ * which keeps being refreshed as the join reader reopens -- or from the indexer ({@link
+ * #resolveFromCalculatedModel}: the pair's edges plus an in-memory {@link JoinColumnModel} that
+ * needs no further indirection to read.
+ */
+ class LeafJoin implements DocEdges {
+ final String pairFieldName;
+ final DocIdSetIterator fromSegmentDocIdIter;
+ final long fromMatchCount;
+ JoinSegmentReference joinSegmentRef;
+ final SegmentsTuple segmentsFromTo;
+ private DocEdges edges;
+ // set only when resolved from the indexer; null means real docvalues are opened through
+ // joinSegmentRef instead
+ private JoinColumnModel docMapping;
+
+ LeafJoin(
+ String pairFieldName,
+ SegmentsTuple segmentsFromTo,
+ DocIdSetIterator fromSegmentDocIdIter,
+ long fromMatchCount) {
+ this.pairFieldName = pairFieldName;
+ this.segmentsFromTo = segmentsFromTo;
+ this.fromSegmentDocIdIter = fromSegmentDocIdIter;
+ this.fromMatchCount = fromMatchCount;
+ assert fromSegmentDocIdIter.docID() != DocIdSetIterator.NO_MORE_DOCS;
+ assert fromSegmentDocIdIter.docID() >= 0;
+ }
+
+ /** Resolves this cell, once, to a join-index-persisted pair's edges. */
+ void resolveFromReadingJoinIndex(DocEdges edges) {
+ assert this.edges == null : "already resolved: " + this.edges;
+ this.edges = edges;
+ }
+
+ /** Resolves this cell, once, to an in-memory doc mapping built on demand. */
+ void resolveFromCalculatedModel(JoinColumnModel docMapping) {
+ assert this.edges == null : "already resolved: " + this.edges;
+ this.edges = docMapping.edges();
+ this.docMapping = docMapping;
+ }
+
+ /**
+ * Binds a freshly built model to this cell, whether it was never resolved or its on-disk column
+ * was reaped from under a running query -- {@link #refreshJoinTasksReferences} rebuilds such a
+ * pair rather than failing the query, and the result lands here.
+ *
+ * <p>Rebinding mid-query is safe because a cell's iteration state lives in {@link
+ * #fromSegmentDocIdIter}, never in the column: {@link #toDocsByFromDocsDV()} is re-opened on
+ * every {@link #dumpMatchesInto} call and read by {@code advanceExact(fromDoc)}, so swapping
+ * what backs it between calls is invisible to the walk. A rebuilt column maps the same
+ * from-docs to the same to-docs as the one it replaces -- pair field names pin both segments --
+ * which the assertion below states.
+ */
+ void bind(JoinColumnModel model) {
+ if (edges == null) {
+ resolveFromCalculatedModel(model);
+ return;
+ }
+ assert Arrays.equals(model.edges().fromDocEdges(), fromDocEdges())
+ && Arrays.equals(model.edges().toDocEdges(), toDocEdges())
+ : "rebuilt column "
+ + pairFieldName
+ + " disagrees with the edges this cell resolved with: "
+ + Arrays.toString(model.edges().fromDocEdges())
+ + "/"
+ + Arrays.toString(model.edges().toDocEdges())
+ + " vs "
+ + Arrays.toString(fromDocEdges())
+ + "/"
+ + Arrays.toString(toDocEdges());
+ this.docMapping = model; // toDocsByFromDocsDV() now bypasses the searcher entirely
+ this.joinSegmentRef = null;
+ rebindsAfterReap++;
+ }
+
+ /**
+ * Points this cell at the column now on disk for its pair and lets go of the model that built
+ * it, so reads go through {@link #joinSegmentRef} like every other cell's.
+ *
+ * <p>A model carries {@code int[fromSegmentMaxDoc]}: 20MB for a five-million-doc from-segment,
+ * per cell, held for as long as the query runs. Eleven live cells in one query and three
+ * queries in flight is how a 2GB heap ends up 74% humongous regions that a full GC cannot
+ * reclaim -- which is what it did. The column on disk says the same thing (it was written from
+ * this very model) and costs an mmap'd docvalues read instead, so the model is worth keeping
+ * only until the sidecar has committed and refreshed past it. The edges stay as they were: the
+ * on-disk ones were written from these.
+ */
+ void adoptOnDiskColumn(JoinSegmentReference onDisk) {
+ assert edges != null : "adopting a column for an unresolved cell: " + this;
+ assert onDisk.pairFieldName().equals(pairFieldName) : onDisk + " is not " + pairFieldName;
+ this.joinSegmentRef = onDisk;
+ this.docMapping = null;
+ }
+
+ boolean isResolved() {
+ return edges != null;
+ }
+
+ SortedNumericDocValues toDocsByFromDocsDV() throws IOException {
+ assert edges != null : "not resolved yet: " + this;
+ if (docMapping != null) {
+ // built on demand, so it's available directly, with no searcher involved
+ return docMapping.toDocByFromDoc();
+ }
+ // resolved from the join index: joinSegmentRef locates the real, on-disk column
+ LeafReaderContext joinContext =
+ lastSeenJoinSearcher.getLeafContexts().get(joinSegmentRef.joinSegmentLeafOrd());
+ assert JoinIndexUtils.segmentName(joinContext).equals(joinSegmentRef.joinSegmentName());
+ return joinContext
+ .reader()
+ .getSortedNumericDocValues(
+ JoinIndexUtils.TO_DOC_VAL_BY_FROM_DOCNUM + joinSegmentRef.pairFieldName());
+ }
+
+ /**
+ * Walks this cell's from-iterator from its current (prepositioned) doc through its edges' last
+ * from-doc, setting every to-doc it maps to -- shifted by {@code shift} -- in {@code
+ * matchedToDocs}.
+ *
+ * @return how many from-docs were walked, i.e. the column-read work this drain cost
+ */
+ int dumpMatchesInto(FixedBitSet matchedToDocs, int shift, int earlyExitDoc) throws IOException {
+ SortedNumericDocValues toDocsByFromDoc = toDocsByFromDocsDV();
+ int walked = 0;
+ boolean confirmedCurrent = false;
+ for (int fromDoc = fromSegmentDocIdIter.docID(); // prepositioned to the first match
+ fromSegIterIsNotExausted();
+ fromDoc = fromSegmentDocIdIter.nextDoc()) {
+ walked++;
+ if (toDocsByFromDoc.advanceExact(
+ fromDoc)) { // TODO will it be faster to use advance() and leapfrog?
+ for (int i = 0; i < toDocsByFromDoc.docValueCount(); i++) {
+ int toDocMatch = (int) toDocsByFromDoc.nextValue();
+ assert toDocMatch <= toDocEdges()[1] && toDocMatch >= toDocEdges()[0]
+ : "to doc "
+ + toDocMatch
+ + " above edges union max "
+ + Arrays.toString(toDocEdges());
+ // shift is wherever the approximation iterator first landed, which need not be
+ // the global firstToDoc (e.g. under a boolean conjunction); a match below shift
+ // is unreachable -- the iterator only moves forward -- so it's dropped rather
+ // than written at a negative offset
+ // Also. we don't need toDocs less than one we're confirming currently
+ if (toDocMatch >= shift && toDocMatch >= earlyExitDoc) {
+ matchedToDocs.set(toDocMatch - shift);
+ if (toDocMatch == earlyExitDoc) {
+ confirmedCurrent = true;
+ }
+ }
+ }
+ if (confirmedCurrent) {
+ // we've just confirmed the doc, which was matches() is called for
+ return walked; // giveup
+ }
+ }
+ }
+ return walked;
+ }
+
+ private boolean fromSegIterIsNotExausted() {
+ return fromSegmentDocIdIter.docID() != DocIdSetIterator.NO_MORE_DOCS
+ && fromSegmentDocIdIter.docID() <= fromDocEdges()[1];
+ }
+
+ @Override
+ public int[] fromDocEdges() {
+ return edges.fromDocEdges();
+ }
+
+ @Override
+ public int[] toDocEdges() {
+ return edges.toDocEdges();
+ }
+
+ @Override
+ public int toCount() {
+ return edges.toCount();
+ }
+ }
+
+ /** Appends {@code cell} to {@link #leafJoins} and registers it in both indices. */
+ private LeafJoin addJoinLeaf(LeafJoin cell) {
+ leafJoins.add(cell);
+ return cell;
+ }
+
+ /**
+ * Drops {@code cell} from {@link #leafJoins}. It's a little bit awkward. Every iteration join
+ * task list is copied and then removed from original list via reference equality. Ideally,
+ * iterators should be used for removals.
+ */
+ private void dropJoinLeaf(LeafJoin cell) {
+ leafJoins.remove(cell);
+ }
+
+ record TaskRefreshResult(
+ Set<Map.Entry<LeafJoin, LeafReaderContext>> joinSegments,
+ Set<Map.Entry<LeafJoin, JoinColumnModel>> justWritten) {}
+
+ /**
+ * @param weightAgeJoinSegmentsReadOnly the join segments cached at {@link
+ * AuxIndexJoinQuery#createWeight} time DON'T MODIFY ME!!!
+ * @param weightAgeJoinSearcher the join searcher cached at {@link AuxIndexJoinQuery#createWeight}
+ * time
+ * @param scorerSupplierAgeJoinSearcher the join searcher used at {@link
+ * JoinIndexWeight#scorerSupplier} time
+ * @param fromColumnFutures from side data by segment ord.
+ */
+ JoinIndexScorerSupplier(
+ LeafReaderContext toContext,
+ String fromField,
+ Query fromQuery,
+ IndexSearcher fromSearcher,
+ String toField,
+ IndexReader toReader,
+ Map<String, JoinSegmentReference> weightAgeJoinSegmentsReadOnly,
+ IndexSearcher weightAgeJoinSearcher,
+ IndexSearcher scorerSupplierAgeJoinSearcher,
+ AuxIndexManager joinIndex,
+ Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures,
+ ScoreMode scoreMode,
+ float boost)
+ throws ExecutionException, InterruptedException, IOException {
+ this.toContext = toContext;
+ this.fromField = fromField;
+ this.fromQuery = fromQuery;
+ this.fromSearcher = fromSearcher;
+ this.toField = toField;
+ this.toReader = toReader;
+ this.fromColumnFutures = fromColumnFutures;
+ this.joinIndex = joinIndex;
+ this.scoreMode = scoreMode;
+ this.boost = boost;
+
+ // 1. check from scorers
+ List<LeafJoin> fromItersTasks = createLeafJoins();
+ for (LeafJoin newJoinTask : fromItersTasks) {
+ this.joinLeafsCreated++;
+ this.addJoinLeaf(newJoinTask);
+ // 2. set old segment references
+ JoinSegmentReference oldReference =
+ weightAgeJoinSegmentsReadOnly.get(newJoinTask.pairFieldName);
+ if (oldReference != null) {
+ newJoinTask.joinSegmentRef = oldReference;
+ }
+ }
+
+ this.lastSeenJoinSearcher =
+ weightAgeJoinSearcher; // set old searcher, it corresponds to weightAgeJoinSegmentsReadOnly
+ TaskRefreshResult refreshedAndNew = refreshJoinTasksReferences(scorerSupplierAgeJoinSearcher);
+ for (Entry<LeafJoin, LeafReaderContext> entry : refreshedAndNew.joinSegments) {
+ LeafJoin task = entry.getKey();
+ LeafReaderContext joinLeaf = entry.getValue(); // got it from searcher leafs by refOrd
+ assert JoinIndexUtils.segmentName(joinLeaf).equals(task.joinSegmentRef.joinSegmentName());
+ assert joinLeaf.ord == task.joinSegmentRef.joinSegmentLeafOrd();
+ task.resolveFromReadingJoinIndex( // ok. this one may be ready for search.
+ new JoinIndexUtils.Edges(
+ JoinIndexUtils.loadEdges(
+ joinLeaf, JoinIndexUtils.FROM_EDGES_PREFIX + task.pairFieldName),
+ // TODO it might not need to be loaded, if "from" edges fully cut of the column. Thus,
+ // we won't read even "to" edges at all.
+ JoinIndexUtils.loadEdges(
+ joinLeaf, JoinIndexUtils.TO_EDGES_PREFIX + task.pairFieldName),
+ // TODO use it for ordering join segment iteration, desc
+ JoinIndexUtils.loadEdges(
+ joinLeaf, JoinIndexUtils.TO_COUNT_PREFIX + task.pairFieldName)[0]));
+ }
+ for (Entry<LeafJoin, JoinColumnModel> entry : refreshedAndNew.justWritten) {
+ LeafJoin task = entry.getKey();
+ task.bind(entry.getValue());
+ }
+ // edges are loaded
+ for (LeafJoin task :
+ List.copyOf(
+ leafJoins)) { // hell. it might remove task from the list. that's sad. I have to copy
+ // it.
+ assert task.isResolved();
+ // a little bit tricky. It assumes that column is join-index backed or just written and
+ // array-backed,
+ advanceAtMinFromEdge(task);
+ }
+ // now let's read each cell's edges, then build "to" side bitset of approximation
+ // first pass: union the contributing pairs' to-doc ranges; every possible match in this
+ // to segment falls into [minToDoc, maxToDoc]
+ for (LeafJoin task : leafJoins) {
+ DocEdges docEdges = task;
+ firstToDoc = Math.min(firstToDoc, docEdges.toDocEdges()[0]);
+ lastToDoc = Math.max(lastToDoc, docEdges.toDocEdges()[1]);
+ matchedToDocsCount += docEdges.toDocEdges()[1] - docEdges.toDocEdges()[0] + 1;
+ if (falsePositiveToDocsBits == null) {
+ falsePositiveToDocsBits = new FixedBitSet(toContext.reader().maxDoc());
+ }
+ falsePositiveToDocsBits.set(docEdges.toDocEdges()[0], docEdges.toDocEdges()[1] + 1);
+ }
+ // when confirming docs, we want to start from heaviest leafs first
+ if (leafJoins.size() > 1) {
+ leafJoins.sort(
+ Comparator.<LeafJoin>comparingInt(
+ leafJoin ->
+ Math.toIntExact(
+ Math.min(
+ leafJoin
+ .fromMatchCount, // use a min of both side as an estimate of join
+ // cardinality
+ leafJoin.edges.toCount())))
+ .reversed());
+ }
+ logContextSetUp();
+ }
+
+ /**
+ * Reports how this context was set up, once. {@code approxCard} is the real size of the
+ * approximation (the union of the surviving pairs' to-ranges), while {@code approxSpanSum} adds
+ * those ranges up with their overlaps counted twice; the two together say how much the
+ * single-range-per-column approximation actually narrows the segment, which is what bounds every
+ * saving the confirmation phase can make.
+ */
+ private void logContextSetUp() {
+ if (!JoinIndexUtils.diagnosticsEnabled(log)) {
+ return;
+ }
+ long colToCountSum = 0;
+ for (LeafJoin cell : leafJoins) {
+ colToCountSum += cell.toCount();
+ }
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=ctx ctx={} toSeg={} toMaxDoc={} cellsCreated={} cellsDroppedApriori={}"
+ + " cellsLive={} buildMs={} approxCard={} approxSpanSum={} approxFrom={} approxTo={}"
+ + " colToCountSum={}",
+ ctxId,
+ JoinIndexUtils.segmentName(toContext),
+ toContext.reader().maxDoc(),
+ joinLeafsCreated,
+ leafsDroppedApriori,
+ leafJoins.size(),
+ joinIndexBuildNanos / 1_000_000L,
+ falsePositiveToDocsBits == null ? 0 : falsePositiveToDocsBits.cardinality(),
+ matchedToDocsCount,
+ firstToDoc,
+ lastToDoc,
+ colToCountSum);
+ }
+
+ /**
+ * Reports the confirmation totals for this context, once. Emitted when every column has been
+ * drained -- i.e. the half-read union has converged and laziness has run out. A context that
+ * never emits this line never had to converge, which is the case the lazy variant exists for.
+ */
+ private void logConfirmationDone(String reason) {
+ if (reported || !JoinIndexUtils.diagnosticsEnabled(log)) {
+ return;
+ }
+ reported = true;
+ JoinIndexUtils.logDiagnostic(
+ log,
+ "AUXIJOIN evt=done ctx={} toSeg={} reason={} confirmCalls={} freeHits={} cellsDrained={}"
+ + " cellsLive={} fromDocsWalked={} rebindsAfterReap={} modelsReleased={} buildMs={}",
+ ctxId,
+ JoinIndexUtils.segmentName(toContext),
+ reason,
+ confirmCalls,
+ confirmFreeHits,
+ leafsDrained,
+ leafJoins.size(),
+ fromDocsWalked,
+ rebindsAfterReap,
+ modelsReleasedToDisk,
+ joinIndexBuildNanos / 1_000_000L);
+ }
+
+ /**
+ * The from-side columns to build {@code tasks}' pairs from, loading on demand any the weight left
+ * out.
+ *
+ * <p>{@code AuxIndexJoinQuery#createWeight} loads the foreign-key column only for from-segments
+ * whose pair was missing from the sidecar -- for a pair that already existed there is nothing to
+ * build, so paying to read its from-side would be waste. A pair reaped from under a running query
+ * turns that on its head: it existed at weight time and has to be rebuilt now, and {@code
+ * JoinColumnIndexer#computeOwnedModels} needs a foreign-key column to rebuild it from. So read it
+ * here, for those from-segments only.
+ *
+ * <p>Returns the weight's own map untouched in the common case, and otherwise a copy: the map is
+ * shared with every other to-segment's supplier of the same weight, and they may be running
+ * concurrently.
+ */
+ private Map<Integer, Future<FromLeafJoinContext>> fromColumnsFor(Collection<LeafJoin> tasks)
+ throws IOException, ExecutionException, InterruptedException {
+ Map<Integer, FromLeafJoinContext> loaded = null;
+ for (LeafJoin task : tasks) {
+ int fromOrd = task.segmentsFromTo.fromLeafOrd();
+ if (loaded != null && loaded.containsKey(fromOrd)) {
+ continue;
+ }
+ FromLeafJoinContext fromContext = fromColumnFutures.get(fromOrd).get();
+ if (fromContext.fkColumn != null) {
+ continue;
+ }
+ if (loaded == null) {
+ loaded = new HashMap<>();
+ }
+ loaded.put(
+ fromOrd,
+ new FromLeafJoinContext(
+ fromContext.matches,
+ new ForeignKeyColumn(fromSearcher.getLeafContexts().get(fromOrd), fromField),
+ fromOrd));
+ }
+ if (loaded == null) {
+ return fromColumnFutures;
+ }
+ Map<Integer, Future<FromLeafJoinContext>> augmented = new HashMap<>(fromColumnFutures);
+ loaded.forEach(
+ (ord, context) -> augmented.put(ord, CompletableFuture.completedFuture(context)));
+ return augmented;
+ }
+
+ private TaskRefreshResult refreshJoinTasksReferences(IndexSearcher newJoinIndexSearcher)
+ throws IOException, ExecutionException, InterruptedException {
+ Set<Map.Entry<LeafJoin, LeafReaderContext>> joinSegments = new LinkedHashSet<>();
+ Set<Map.Entry<LeafJoin, JoinColumnModel>> justWritten = new LinkedHashSet<>();
+
+ Set<LeafJoin> refreshReference = new LinkedHashSet<>();
+ Set<LeafJoin> loadReference = new LinkedHashSet<>();
+ Map<String, LeafJoin> needIndex = new LinkedHashMap<>();
+ // cells still reading the model they were built from; this refresh is their chance to let it go
+ Map<String, LeafJoin> modelBacked = new LinkedHashMap<>();
+ Set<LeafJoin> resolveTarget =
+ (Objects.equals(newJoinIndexSearcher, this.lastSeenJoinSearcher))
+ ? loadReference
+ : refreshReference;
+ for (LeafJoin task : leafJoins) {
+ if (task.joinSegmentRef != null) {
+ resolveTarget.add(task);
+ } else if (!task.isResolved()) {
+ needIndex.put(task.pairFieldName, task);
+ } else {
+ modelBacked.put(task.pairFieldName, task);
+ }
+ }
+ // from here on every reference has to address newJoinIndexSearcher: it is the searcher this
+ // supplier reads its columns from (see LeafJoin#toDocsByFromDocsDV, which resolves a reference
+ // against lastSeenJoinSearcher by leaf ord), so passes 2 and 3 below must look segments up in
+ // it and not in the one the weight happened to see. They used to scan the older searcher,
+ // which went unnoticed while the sidecar only ever appended segments and dropped whole dead
+ // ones; a doc-aligned compaction (AuxIndexJoinMergePolicy) reshuffles leaf ords under a live
+ // query, and then a reference resolved in the old searcher addresses a different segment here.
+ this.lastSeenJoinSearcher = newJoinIndexSearcher;
+ // pass 0: hand back the models. A cell built during this query reads the model it was built
+ // from until the sidecar has committed and refreshed past the batch -- which is exactly what
+ // this searcher, acquired after that write, proves. Adopting the column releases an
+ // int[fromSegmentMaxDoc] per cell, mid-query rather than at the end of it, and from here on
+ // the cell is an ordinary reference-backed one that passes 1 to 3 keep pointing at the right
+ // segment. A column not found is one the reaper dropped between the write and now, or one the
+ // sidecar has not refreshed past because blockingRefresh is off: the model stays as it is, and
+ // the next refresh tries again.
+ int modelsReleased = 0;
+ if (!modelBacked.isEmpty()) {
+ for (Map.Entry<String, JoinSegmentReference> onDisk :
+ JoinIndexUtils.extractExistingJoinColumns(newJoinIndexSearcher, modelBacked::containsKey)
+ .entrySet()) {
+ modelBacked.get(onDisk.getKey()).adoptOnDiskColumn(onDisk.getValue());
+ modelsReleased++;
+ }
+ this.modelsReleasedToDisk += modelsReleased;
+ }
+ // refresh old refs, pass 1: same searcher, just get a segment by ord and check
+ // the segment name
+ List<LeafReaderContext> newLeaves = newJoinIndexSearcher.getLeafContexts();
+ for (Iterator<LeafJoin> iter = refreshReference.iterator(); iter.hasNext(); ) {
+ LeafJoin task = iter.next();
+ // check segment name by ord, if true, resolve it right here, remove from here
+ LeafReaderContext byOrd =
+ task.joinSegmentRef.joinSegmentLeafOrd() < newLeaves.size()
+ ? newLeaves.get(task.joinSegmentRef.joinSegmentLeafOrd())
+ : null;
+ if (byOrd != null
+ && JoinIndexUtils.segmentName(byOrd).equals(task.joinSegmentRef.joinSegmentName())) {
+ joinSegments.add(new SimpleEntry<>(task, byOrd));
+ iter.remove();
+ }
+ }
+ // pass 2: searcher have changed, need to lookup segments by name in the new one
+ if (!refreshReference.isEmpty()) {
+ Map<String, LeafJoin> byOldJoinSegName = new HashMap<>();
+ for (LeafJoin task : refreshReference) {
+ byOldJoinSegName.put(task.joinSegmentRef.joinSegmentName(), task);
+ }
+ for (LeafReaderContext joinLeaf : newLeaves) {
+ String segName = JoinIndexUtils.segmentName(joinLeaf);
+ LeafJoin task = byOldJoinSegName.get(segName);
+ if (task != null) {
+ task.joinSegmentRef =
+ new JoinSegmentReference(task.joinSegmentRef.pairFieldName(), segName, joinLeaf.ord);
+ joinSegments.add(new SimpleEntry<>(task, joinLeaf));
+ refreshReference.remove(task);
+ }
+ }
+ }
+ // pass 3: search by field name
+ if (!refreshReference.isEmpty()) {
+ Map<String, LeafJoin> byPairFieldName = new HashMap<>();
+ for (LeafJoin task : refreshReference) {
+ byPairFieldName.put(task.joinSegmentRef.pairFieldName(), task);
+ }
+ // loop join segments search for fields
+ Map<String, JoinSegmentReference> joinSegmentsByPairFieldName =
+ JoinIndexUtils.extractExistingJoinColumns(
+ lastSeenJoinSearcher, byPairFieldName::containsKey);
+ // if found move to load set
+ for (LeafJoin task : byPairFieldName.values()) {
+ JoinSegmentReference found = joinSegmentsByPairFieldName.get(task.pairFieldName);
+ if (found != null) {
+ task.joinSegmentRef = found;
+ loadReference.add(task);
+ refreshReference.remove(task);
+ }
+ }
+ }
+ // pass 4: the column is gone from the sidecar entirely -- the reaper dropped it while this
+ // query was running (AuxIndexJoinMergePolicy queues a pair the moment a sample stops needing
+ // it, which can race a query that still does). Rebuild it rather than failing the query: the
+ // pair name pins both segments, so what comes back maps exactly what was lost. Reads through an
+ // already-acquired searcher never land here -- that reader pins its files -- only a refresh
+ // onto a newer searcher does.
+ for (LeafJoin task : refreshReference) {
+ needIndex.put(task.pairFieldName, task);
+ }
+ refreshReference.clear();
+ // load edges for regulars, repeat pass 1, for those who was found at pass 3
+ for (LeafJoin cell : loadReference) {
+ // String pairFieldName = cell.pairFieldName;
+ LeafReaderContext joinLeafSeg =
+ lastSeenJoinSearcher.getLeafContexts().get(cell.joinSegmentRef.joinSegmentLeafOrd());
+ assert JoinIndexUtils.segmentName(joinLeafSeg).equals(cell.joinSegmentRef.joinSegmentName());
+ joinSegments.add(new SimpleEntry<>(cell, joinLeafSeg));
+ }
+ // index unlucky ones
+ if (!needIndex.isEmpty()) {
+
+ Map<String, SegmentsTuple> missingPairs = new HashMap<>();
+ for (LeafJoin cell : needIndex.values()) {
+ missingPairs.put(cell.pairFieldName, cell.segmentsFromTo);
+ }
+ long buildStartNanos = System.nanoTime();
+ Map<String, JoinColumnModel> written =
+ this.joinIndex.buildAndPersistJoinColumns(
+ Collections.unmodifiableMap(missingPairs),
+ this.fromSearcher.getIndexReader(),
+ this.toReader,
+ this.toField,
+ this.ctxId,
+ // the searcher these pairs were last seen absent in: null refs trace back to the
+ // weight-age extract, and pass 3 above rescanned newJoinIndexSearcher for them,
+ // which lastSeenJoinSearcher now is -- the freshest absence this supplier observed
+ this.lastSeenJoinSearcher,
+ fromColumnsFor(needIndex.values()));
+ this.joinIndexBuildNanos += System.nanoTime() - buildStartNanos;
+ assert written.keySet().containsAll(missingPairs.keySet());
+ assert missingPairs.keySet().containsAll(written.keySet());
+ for (Map.Entry<String, JoinColumnModel> entry : written.entrySet()) { // TODO optimize
+ LeafJoin cell = needIndex.get(entry.getKey());
+ justWritten.add(new SimpleEntry<>(cell, entry.getValue()));
+ }
+ }
+ assert Objects.equals(this.lastSeenJoinSearcher, newJoinIndexSearcher);
+ return new TaskRefreshResult(joinSegments, justWritten);
+ }
+
+ /**
+ * Positions {@code cell}'s from-iterator behind {@code docEdges}'s first from-doc, or drops
+ * {@code cell} via {@link #dropJoinLeaf} when the iterator can no longer reach any doc the pair
+ * maps -- either because the pair maps nothing ({-1, -1} sentinel), the iterator already moved
+ * past the pair's last from-doc, or it exhausts before reaching the pair's first one (the
+ * iterator only moves forward, so none of these are recoverable). Returns whether the cell
+ * survives.
+ */
+ private boolean advanceAtMinFromEdge(LeafJoin cell) throws IOException {
+ int[] fromDocEdges = cell.fromDocEdges();
+ int minFromDoc = fromDocEdges[0];
+ int maxFromDoc = fromDocEdges[1];
+ DocIdSetIterator fromSegemtIter = cell.fromSegmentDocIdIter;
+ if (minFromDoc < 0) {
+ // {-1, -1} sentinel: this pair maps no from doc to any to doc at all
+ leafsDroppedApriori++;
+ dropJoinLeaf(cell);
+ return false;
+ }
+ if (maxFromDoc >= 0
+ && maxFromDoc != DocIdSetIterator.NO_MORE_DOCS
+ && fromSegemtIter.docID() > maxFromDoc) {
+ // from iter is already past the last from doc this pair maps, so it cannot contribute
+ leafsDroppedApriori++;
+ dropJoinLeaf(cell); // no more matches in this join segment, so the pair cannot contribute
+ return false;
+ }
+ if (minFromDoc >= 0
+ && maxFromDoc != DocIdSetIterator.NO_MORE_DOCS
+ && fromSegemtIter.docID() < minFromDoc) {
+ int firstMatch = fromSegemtIter.advance(minFromDoc);
+ if (firstMatch == DocIdSetIterator.NO_MORE_DOCS || firstMatch > maxFromDoc) {
+ /// wow from iter exhausted, no match in this join segment, so the pair cannot contribute
+ // thus we need to return them from request `
+ leafsDroppedApriori++;
+ dropJoinLeaf(cell); // no more matches in this join segment, so the pair cannot contribute
+ return false;
+ } // else from iter is advanced behind the first from match , good
+ }
+ return true;
+ }
+
+ /**
+ * every to segment call for prepositioned from seg iters populates a {@link LeafJoin} per
+ * contributing from segment, its iterator PREPOSITIONED to the first matching doc
+ *
+ * @return tasks are orfered by descending from-side match count, so the first task is the one
+ * with the most matches
+ */
+ private List<LeafJoin> createLeafJoins()
+ throws ExecutionException, InterruptedException, IOException {
+ List<LeafJoin> tasks = new ArrayList<>();
+ for (Future<FromLeafJoinContext> fromContextFuture : this.fromColumnFutures.values()) {
+ // block on them in the given order, expecting to park on the longest first, and then read
+ // shorter with no locks
+ FromLeafJoinContext matchAndCount = fromContextFuture.get();
+ if (matchAndCount == null) { // no hits for this from-side leaf.
+ continue;
+ }
+ LeafReaderContext fromContext = this.fromSearcher.getLeafContexts().get(matchAndCount.ord);
+ DocIdSetIterator matchedFromDocs;
+ if (matchAndCount.matches != null
+ && (matchedFromDocs = matchAndCount.matches.iterator()) != null
+ && matchedFromDocs.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
+ // name every contributing (from, to) pair column; pair field names are unique across pairs
+ String pairFieldName =
+ JoinIndexUtils.pairFieldName(fromContext, this.fromField, toContext, this.toField);
+ tasks.add(
+ new LeafJoin(
+ pairFieldName,
+ new SegmentsTuple(fromContext.ord, toContext.ord),
+ matchedFromDocs,
+ matchAndCount.matches.count()));
+ }
+ }
+ return tasks;
+ }
+
+ /** True if this context has any candidate docs at all; used by {@code JoinIndexWeight}. */
+ boolean isEmpty() {
+ return falsePositiveToDocsBits == null || matchedToDocsCount == 0;
+ }
+
+ @Override
+ public Scorer get(long leadCost) throws IOException {
+ assert !isEmpty();
+ DocIdSetIterator approximation =
+ new BitSetIterator(falsePositiveToDocsBits, matchedToDocsCount);
+
+ TwoPhaseIterator twoPhase = new LazyConfirmationIterator(approximation);
+ return new ConstantScoreScorer(boost, scoreMode, twoPhase);
+ }
+
+ @Override
+ public long cost() {
+ return (long) lastToDoc - firstToDoc + 1;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexUtils.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexUtils.java
new file mode 100644
index 00000000000..bc1114c39e1
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexUtils.java
@@ -0,0 +1,1061 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import org.apache.lucene.codecs.Codec;
+import org.apache.lucene.codecs.FieldInfosFormat;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.FilterCodecReader;
+import org.apache.lucene.index.FilterLeafReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ParallelCompositeReader;
+import org.apache.lucene.index.ParallelLeafReader;
+import org.apache.lucene.index.SegmentCommitInfo;
+import org.apache.lucene.index.SegmentReader;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.search.BulkScorer;
+import org.apache.lucene.search.DocIdSet;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.DocIdStream;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.LeafCollector;
+import org.apache.lucene.search.Scorable;
+import org.apache.lucene.search.ScorerSupplier;
+import org.apache.lucene.search.Weight;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FSDirectory;
+import org.apache.lucene.store.FilterDirectory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.BitDocIdSet;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.CollectionUtil;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.RoaringDocIdSet;
+import org.apache.lucene.util.StringHelper;
+import org.apache.lucene.util.packed.PackedInts;
+import org.apache.lucene.util.packed.PackedLongValues;
+import org.apache.lucene.util.packed.PagedMutable;
+import org.slf4j.Logger;
+import org.slf4j.event.Level;
+
+/**
+ * Column-building and addressing helpers for the auxiliary join index managed by {@link
+ * AuxIndexManager}: for every (from-segment, to-segment) pair it produces a SORTED_NUMERIC column
+ * named {@link #pairFieldName}, whose doc number is the from-side doc id and whose value is the
+ * to-side doc id whose {@code toField} term equals the from doc's {@code fromField} term, plus two
+ * companion edges columns persisting the pair's {min, max} from-doc and to-doc bounds.
+ *
+ * @lucene.experimental
+ */
+final class JoinIndexUtils {
+
+ /** Suffix of the always-written column persisting a pair's {min, max} from-doc edges. */
+ static final String FROM_EDGES_PREFIX = "fromDoc_edges_"; // TODO reduce to the singe letter
+
+ /** Suffix of the always-written column persisting a pair's {min, max} to-doc edges. */
+ static final String TO_EDGES_PREFIX = "toDoc_edges_";
+
+ /** main join columns for join index to_doc_num[from_docnum] */
+ static final String TO_DOC_VAL_BY_FROM_DOCNUM = "join_toDoc_";
+
+ static final String TO_COUNT_PREFIX = "num_toDoc_";
+
+ /**
+ * Every field one pair occupies in a sidecar segment -- the inverse of {@link #pairFieldNames},
+ * which recovers pair names from field names. Reaping a pair means hiding exactly these; keep the
+ * two in step if a fifth column is ever added.
+ */
+ static Set<String> columnFieldNames(String pairFieldName) {
+ return Set.of(
+ TO_DOC_VAL_BY_FROM_DOCNUM + pairFieldName,
+ FROM_EDGES_PREFIX + pairFieldName,
+ TO_EDGES_PREFIX + pairFieldName,
+ TO_COUNT_PREFIX + pairFieldName);
+ }
+
+ /**
+ * Commit user-data key under which the {@code SolrVersion} that wrote the index is recorded, so a
+ * future major version can detect and discard an incompatible aux join index.
+ */
+ static final String AUX_INDEX_VERSION = "AuxJoinIndexVersion";
+
+ /**
+ * Values per page in every paged structure this join builds -- columns, the from-side maps, the
+ * to-side inversion. Sized so a page stays comfortably under G1's humongous threshold even in the
+ * worst case: 16384 values at the full 64 bits is 128KB, against half of a 1MB region.
+ */
+ static final int PAGE_SIZE = 1 << 14;
+
+ private JoinIndexUtils() {}
+
+ /**
+ * Level the {@code AIJOIN evt=...} diagnostic logs emit at; kept in one place so all diagnostics
+ * can be raised or lowered together. To see them, enable {@code TRACE} for the emitting loggers
+ * in log4j config.
+ */
+ private static final Level LOG_LEVEL = Level.TRACE;
+
+ /** Whether the AIJOIN diagnostic logs would emit at the configured level. */
+ static boolean diagnosticsEnabled(Logger log) {
+ return log.isEnabledForLevel(LOG_LEVEL);
+ }
+
+ /** Emits an AIJOIN diagnostic line at the configured level. */
+ static void logDiagnostic(Logger log, String message, Object... args) {
+ log.atLevel(LOG_LEVEL).log(message, args);
+ }
+
+ /**
+ * Scans {@code joinSearcher}'s leaves for every pair column whose field name satisfies {@code
+ * isNeeded}, returning where each one lives. Used both to seed a fresh {@link JoinIndexWeight}'s
+ * view of already-built pairs, and by {@link JoinIndexScorerSupplier} to relocate a pair whose
+ * cached segment reference no longer resolves. TODO subject for in-heap caching TODO commit's
+ * userdata might have a list of pairs with known segment ords and names
+ */
+ static Map<String, AuxIndexManager.JoinSegmentReference> extractExistingJoinColumns(
+ IndexSearcher joinSearcher, Predicate<String> isNeeded) {
+ Map<String, AuxIndexManager.JoinSegmentReference> existingJoinSegments =
+ CollectionUtil.newHashMap(joinSearcher.getIndexReader().leaves().size());
+ for (LeafReaderContext joinContext : joinSearcher.getIndexReader().leaves()) {
+ String segmentName = segmentName(joinContext);
+ for (FieldInfo fieldInfo : joinContext.reader().getFieldInfos()) {
+ // pairs are detected by their toCount companion, which is written to doc 0 for every
+ // built pair; the join column itself is sparse and a tombstone pair (disjoint terms)
+ // never materializes it, so scanning for join columns kept re-reporting once-built
+ // tombstones as missing -- and re-triggering their from-side FK loads on every query
+ String splits[] = fieldInfo.name.split(TO_COUNT_PREFIX);
+ if (splits.length == 2 && isNeeded.test(splits[1])) {
+ existingJoinSegments.computeIfAbsent(
+ splits[1],
+ fieldName ->
+ new AuxIndexManager.JoinSegmentReference(
+ fieldName, segmentName, joinContext.ord));
+ }
+ }
+ }
+ return existingJoinSegments;
+ }
+
+ static CacheAndCount computeDocIdSet(Weight fromWeight, LeafReaderContext ctx)
+ throws IOException {
+ ScorerSupplier supplier = fromWeight.scorerSupplier(ctx);
+ if (supplier == null) {
+ return null; // NO matches ???
+ }
+ BulkScorer scorer = supplier.bulkScorer();
+ return cacheImpl(scorer, ctx.reader().maxDoc(), ctx.reader().getLiveDocs());
+ }
+
+ /**
+ * A pair's {min, max} from-doc and to-doc bounds and match count, common to both a pair freshly
+ * built on demand ({@link JoinColumnModel#edges()}) and one already persisted in the join index
+ * ({@link Edges}, loaded through {@link #loadEdges}), so code walking matches doesn't need to
+ * care which one backs it.
+ */
+ interface DocEdges {
+ int[] fromDocEdges();
+
+ int[] toDocEdges();
+
+ /** this is rather doubtful */
+ int toCount();
+ }
+
+ /** A self-contained {@link DocEdges} value, with no addressing information of its own. */
+ record Edges(int[] fromDocEdges, int[] toDocEdges, int toCount) implements DocEdges {}
+
+ /**
+ * The from-doc -> to-doc map produced by {@link #computeDocMapping}, paired with its resolved
+ * {@link #edges()}. {@link #toDocByFromDoc()} mirrors the on-disk column's read API, so freshly
+ * built pairs (not yet flushed to the join index) and pairs loaded from the join index can be
+ * walked by the same code.
+ *
+ * <p>Two layouts back that read API, chosen per pair by {@link #sparseIsSmaller} from the match
+ * count the measuring pass established:
+ *
+ * <ul>
+ * <li><b>dense</b> -- one slot per from-doc up to the pair's last match, holding {@code toDoc +
+ * 1} so that {@code 0} means "no match here". Addressed straight by from-doc, so a probe is
+ * one read.
+ * <li><b>sparse</b> -- only the from-docs that do have a match, ascending, alongside the to-doc
+ * each maps to. A probe searches the from-docs, which a cursor keeps linear by resuming
+ * where the previous one stopped.
+ * </ul>
+ *
+ * <p>Sparse is the smaller of the two below 50% density, and the layouts are far from evenly
+ * used: over 34k column drains from a production run, 71% of the columns matched under 1% of
+ * their from-segment's docs (median 2,125 of 5,031,945) while the other 29% matched essentially
+ * every doc.
+ *
+ * <p>Both layouts are held as {@link PackedLongValues}, which stores values at the width they
+ * need rather than a flat 32 bits, and pages them at {@link #PAGE_SIZE} rather than in one array.
+ * Paging is the point: a five-million-doc from-segment as one {@code int[]} is 20MB, and G1 gives
+ * any allocation over half a region (512KB at a 1MB region size) its own run of regions, so
+ * columns like that had taken a production heap 74% humongous -- unreclaimable by anything short
+ * of a full collection, which is how this crashed. No page here reaches 128KB whatever the
+ * column's width or density, so no column is ever allocated humongous again; the narrower packing
+ * (23 bits for five million to-docs) is a second, smaller win on top.
+ */
+ static final class JoinColumnModel {
+
+ private final int maxDoc;
+ private final DocEdges edges;
+
+ /**
+ * Dense layout: {@code toDoc + 1} at each from-doc, {@code 0} where the from-doc has no match,
+ * running to the last matched from-doc and no further. Null when sparse.
+ */
+ private final PackedLongValues toDocPlusOneByFromDoc;
+
+ /** Sparse layout: the from-docs carrying a match, ascending. Null when dense. */
+ private final PackedLongValues matchedFromDocs;
+
+ /** Sparse layout: the to-doc of the from-doc at the same index. Null when dense. */
+ private final PackedLongValues matchedToDocs;
+
+ private JoinColumnModel(
+ int maxDoc,
+ DocEdges edges,
+ PackedLongValues toDocPlusOneByFromDoc,
+ PackedLongValues matchedFromDocs,
+ PackedLongValues matchedToDocs) {
+ this.maxDoc = maxDoc;
+ this.edges = edges;
+ this.toDocPlusOneByFromDoc = toDocPlusOneByFromDoc;
+ this.matchedFromDocs = matchedFromDocs;
+ this.matchedToDocs = matchedToDocs;
+ }
+
+ /**
+ * Collects one pair's matches into the layout {@code sparse} selects. Matches must arrive in
+ * ascending from-doc order, which is how both build passes walk the from-segment; the dense
+ * layout relies on it to know how many empty slots to lay down before each match.
+ */
+ static final class Builder {
+ private final int maxDoc;
+ private final boolean sparse;
+ private final PackedLongValues.Builder matchedFromDocs;
+ private final PackedLongValues.Builder toDocs;
+
+ /** Dense layout: how many slots are already laid down, i.e. the next from-doc to fill. */
+ private int nextSlot;
+
+ private int matches;
+
+ Builder(int maxDoc, boolean sparse) {
+ this.maxDoc = maxDoc;
+ this.sparse = sparse;
+ // from-docs only ever ascend, and the monotonic encoding stores each one as its distance
+ // from a linear approximation of the whole column -- so a nearly-dense column, whose
+ // from-docs step by one, costs about nothing to remember, which is exactly the column
+ // whose to-docs are most expensive.
+ this.matchedFromDocs =
+ sparse ? PackedLongValues.monotonicBuilder(PAGE_SIZE, PackedInts.COMPACT) : null;
+ this.toDocs = PackedLongValues.packedBuilder(PAGE_SIZE, PackedInts.COMPACT);
+ }
+
+ /** Records that {@code fromDoc} matches {@code toDoc}. */
+ void add(int fromDoc, int toDoc) {
+ assert fromDoc >= nextSlot : "matches must ascend, got " + fromDoc + " after " + nextSlot;
+ assert toDoc >= 0 : "a match must name a to-doc, got " + toDoc;
+ if (sparse) {
+ matchedFromDocs.add(fromDoc);
+ toDocs.add(toDoc);
+ } else {
+ while (nextSlot < fromDoc) {
+ toDocs.add(0L); // no match at this from-doc
+ nextSlot++;
+ }
+ toDocs.add(toDoc + 1L);
+ }
+ nextSlot = fromDoc + 1;
+ matches++;
+ }
+
+ /** How many matches were added, i.e. what {@code edges.toCount()} has to be. */
+ int matches() {
+ return matches;
+ }
+
+ JoinColumnModel build(DocEdges edges) {
+ assert matches == edges.toCount() : "added " + matches + " matches for edges " + edges;
+ return sparse
+ ? new JoinColumnModel(maxDoc, edges, null, matchedFromDocs.build(), toDocs.build())
+ : new JoinColumnModel(maxDoc, edges, toDocs.build(), null, null);
+ }
+ }
+
+ /** A builder laying out a {@code maxDoc}-wide pair densely or sparsely. */
+ static Builder builder(int maxDoc, boolean sparse) {
+ return new Builder(maxDoc, sparse);
+ }
+
+ /** A model laid out densely from {@code toDocByFromDoc}, {@code -1} meaning no match. */
+ static JoinColumnModel dense(int[] toDocByFromDoc, DocEdges edges) {
+ return of(toDocByFromDoc, edges, false);
+ }
+
+ /** The same map as {@link #dense}, laid out as its matches only. */
+ static JoinColumnModel sparse(int[] toDocByFromDoc, DocEdges edges) {
+ return of(toDocByFromDoc, edges, true);
+ }
+
+ private static JoinColumnModel of(int[] toDocByFromDoc, DocEdges edges, boolean sparse) {
+ Builder builder = builder(toDocByFromDoc.length, sparse);
+ for (int fromDoc = 0; fromDoc < toDocByFromDoc.length; fromDoc++) {
+ if (toDocByFromDoc[fromDoc] >= 0) {
+ builder.add(fromDoc, toDocByFromDoc[fromDoc]);
+ }
+ }
+ return builder.build(edges);
+ }
+
+ /**
+ * A model for a pair that matches nothing, holding no per-doc storage at all. It still reports
+ * {@code maxDoc}, so a batch made only of tombstones is still sized to the from-segments it
+ * stands for, and its edges are the symmetric {@code {-1, -1}} sentinel: an asymmetric one
+ * (e.g. {@code {NO_MORE_DOCS, -1}}) doesn't round-trip through the join index's SORTED_NUMERIC
+ * edges column, which always returns its two values in ascending numeric order regardless of
+ * which was written as "min".
+ */
+ static JoinColumnModel tombstone(int maxDoc) {
+ return builder(maxDoc, true).build(new Edges(new int[] {-1, -1}, new int[] {-1, -1}, 0));
+ }
+
+ /**
+ * Whether {@code toCount} matches spread over {@code denseSlots} from-docs are cheaper held as
+ * matches than as a slot per from-doc -- i.e. whether the column is under 50% dense. Sparse
+ * remembers two values per match against dense's one per slot, and while its from-doc column
+ * actually costs far less than that (see {@link Builder}), counting it at full price keeps the
+ * choice a comparison the measuring pass can make with no second guess about how well anything
+ * will pack.
+ */
+ static boolean sparseIsSmaller(int toCount, int denseSlots) {
+ return (long) toCount * 2 < denseSlots;
+ }
+
+ /**
+ * Returns a fresh single-valued cursor over the from-doc -> to-doc map, positioned before doc
+ * 0. Both cursors only move forward, which is what the on-disk columns require of their readers
+ * anyway; see {@link SparseColumnValues}.
+ */
+ SortedNumericDocValues toDocByFromDoc() {
+ return toDocPlusOneByFromDoc != null
+ ? new DenseColumnValues(toDocPlusOneByFromDoc)
+ : new SparseColumnValues(matchedFromDocs, matchedToDocs);
+ }
+
+ DocEdges edges() {
+ return edges;
+ }
+
+ /** Returns one greater than the largest possible document number. */
+ public int maxDoc() {
+ return maxDoc;
+ }
+
+ /** Whether this model holds only its matches, rather than a slot per from-doc. */
+ boolean isSparse() {
+ return toDocPlusOneByFromDoc == null;
+ }
+
+ /** What this model's per-doc storage costs on the heap, for the build diagnostics. */
+ long ramBytesUsed() {
+ return isSparse()
+ ? matchedFromDocs.ramBytesUsed() + matchedToDocs.ramBytesUsed()
+ : toDocPlusOneByFromDoc.ramBytesUsed();
+ }
+ }
+
+ /**
+ * The dense layout's cursor: {@code toDoc + 1} addressed straight by from-doc, {@code 0} where
+ * there is no match, so a probe is a single read and needs no search. Slots past the pair's last
+ * match were never laid down, which {@link #size} stands for. Always single-valued until M:N
+ * pairs are supported.
+ */
+ private static final class DenseColumnValues extends SortedNumericDocValues {
+ private final PackedLongValues toDocPlusOneByFromDoc;
+ private final int size;
+ private int doc = -1;
+
+ /** The to-doc at {@link #doc}, read by the same probe that found it. */
+ private long toDoc;
+
+ DenseColumnValues(PackedLongValues toDocPlusOneByFromDoc) {
+ this.toDocPlusOneByFromDoc = toDocPlusOneByFromDoc;
+ this.size = Math.toIntExact(toDocPlusOneByFromDoc.size());
+ }
+
+ @Override
+ public long nextValue() {
+ return toDoc;
+ }
+
+ @Override
+ public int docValueCount() {
+ return 1;
+ }
+
+ @Override
+ public boolean advanceExact(int target) {
+ doc = target;
+ if (target >= size) {
+ return false;
+ }
+ long toDocPlusOne = toDocPlusOneByFromDoc.get(target);
+ toDoc = toDocPlusOne - 1;
+ return toDocPlusOne != 0;
+ }
+
+ @Override
+ public int docID() {
+ return doc;
+ }
+
+ @Override
+ public int nextDoc() {
+ return doc == NO_MORE_DOCS ? NO_MORE_DOCS : advance(doc + 1);
+ }
+
+ @Override
+ public int advance(int target) {
+ for (int fromDoc = target; fromDoc < size; fromDoc++) {
+ long toDocPlusOne = toDocPlusOneByFromDoc.get(fromDoc);
+ if (toDocPlusOne != 0) {
+ toDoc = toDocPlusOne - 1;
+ doc = fromDoc;
+ return doc;
+ }
+ }
+ doc = NO_MORE_DOCS;
+ return doc;
+ }
+
+ @Override
+ public long cost() {
+ return size;
+ }
+ }
+
+ /**
+ * The sparse layout's cursor: the same read API over the matched from-docs (ascending, no
+ * duplicates while the mapping stays single-valued) and their to-docs.
+ *
+ * <p>Unlike the dense cursor, which is addressed by from-doc directly, this one has to find the
+ * from-doc -- so it only moves forward, resuming each seek where the previous one stopped, which
+ * keeps a full walk of the column linear instead of a binary search per doc. That is the standard
+ * {@link org.apache.lucene.index.DocValuesIterator} contract, and the one the on-disk columns
+ * already impose on the same call sites; both of them here -- {@code
+ * JoinColumnDocWriter.PairColumn} walking {@code nextDoc()} up the batch, and {@code
+ * JoinIndexScorerSupplier.LeafJoin dumpMatchesInto} calling {@code advanceExact(fromDoc)} along a
+ * forward-only from-doc iterator -- take a freshly positioned cursor and only ever move it
+ * forward.
+ */
+ private static final class SparseColumnValues extends SortedNumericDocValues {
+ private final PackedLongValues matchedFromDocs;
+ private final PackedLongValues matchedToDocs;
+ private final int size;
+
+ /** Where the last seek stopped: the first match at or after the doc last asked for. */
+ private int index;
+
+ /** The from-doc at {@link #index}, or {@code NO_MORE_DOCS} once the column is spent. */
+ private int indexedFromDoc;
+
+ private int doc = -1;
+
+ SparseColumnValues(PackedLongValues matchedFromDocs, PackedLongValues matchedToDocs) {
+ this.matchedFromDocs = matchedFromDocs;
+ this.matchedToDocs = matchedToDocs;
+ this.size = Math.toIntExact(matchedFromDocs.size());
+ }
+
+ @Override
+ public long nextValue() {
+ return matchedToDocs.get(index);
+ }
+
+ @Override
+ public int docValueCount() {
+ return 1;
+ }
+
+ @Override
+ public int docID() {
+ return doc;
+ }
+
+ @Override
+ public boolean advanceExact(int target) {
+ seek(target);
+ doc = target;
+ return indexedFromDoc == target;
+ }
+
+ @Override
+ public int nextDoc() {
+ return doc == NO_MORE_DOCS ? NO_MORE_DOCS : advance(doc + 1);
+ }
+
+ @Override
+ public int advance(int target) {
+ seek(target);
+ doc = indexedFromDoc;
+ return doc;
+ }
+
+ /**
+ * Positions {@link #index} on the first match at or after {@code target}, and {@link
+ * #indexedFromDoc} on its from-doc. Matches before the cursor can't answer a target that never
+ * goes backwards, so the search starts there: stepping to the next match or two -- how a full
+ * walk of the column advances -- costs a read, and only a real jump falls through to a binary
+ * search over what is left.
+ */
+ private void seek(int target) {
+ assert doc == NO_MORE_DOCS || target >= doc
+ : "sparse cursor only moves forward, got " + target + " at " + doc;
+ int at = index;
+ for (int probe = 0; probe < 2 && at < size; probe++, at++) {
+ int fromDoc = (int) matchedFromDocs.get(at);
+ if (fromDoc >= target) {
+ index = at;
+ indexedFromDoc = fromDoc;
+ return;
+ }
+ }
+ int low = at;
+ int high = size - 1;
+ while (low <= high) {
+ int mid = (low + high) >>> 1;
+ int fromDoc = (int) matchedFromDocs.get(mid);
+ if (fromDoc < target) {
+ low = mid + 1;
+ } else if (fromDoc > target) {
+ high = mid - 1;
+ } else {
+ index = mid;
+ indexedFromDoc = fromDoc;
+ return;
+ }
+ }
+ index = low;
+ indexedFromDoc = low < size ? (int) matchedFromDocs.get(low) : NO_MORE_DOCS;
+ }
+
+ @Override
+ public long cost() {
+ return size;
+ }
+ }
+
+ /**
+ * Maps a to-side {@link LeafReaderContext} to the live to-side doc id at each of its ordinals,
+ * for a fixed {@code toField}. this is quite local lifecycle class, thus we can cache it so. the
+ * trick is that, there always single entry in this cache.
+ *
+ * <p>The inversion is addressed by ordinal in no particular order, so unlike a column it has to
+ * stay mutable -- but it is still paged, at {@link #PAGE_SIZE}, and packed at the bits a to-doc
+ * needs rather than a flat 32: a five-million-doc to-segment inverted to one {@code int[]} was a
+ * 20MB humongous allocation per to-segment loaded.
+ */
+ static class ToDocInvertor implements Function<LeafReaderContext, PagedMutable> {
+ private final String toField;
+ private final Map<Integer, PagedMutable> cache = new HashMap<>();
+
+ ToDocInvertor(String toField) {
+ this.toField = toField;
+ }
+
+ @Override
+ public PagedMutable apply(LeafReaderContext toContext) {
+ return cache.computeIfAbsent(toContext.ord, (i) -> computeToDoc(toContext));
+ }
+
+ /** {@code toDoc + 1} at each to-ord, {@code 0} where no live to-doc carries that ordinal. */
+ private PagedMutable computeToDoc(LeafReaderContext toContext) {
+ try {
+ SortedSetDocValues toDV = DocValues.getSortedSet(toContext.reader(), toField);
+ Bits toLiveDocs = toContext.reader().getLiveDocs();
+ PagedMutable toDocPlusOneByToOrd =
+ new PagedMutable(
+ toDV.getValueCount(),
+ PAGE_SIZE,
+ PackedInts.bitsRequired(toContext.reader().maxDoc()),
+ PackedInts.COMPACT);
+ for (int toDoc = toDV.nextDoc();
+ toDoc != DocIdSetIterator.NO_MORE_DOCS;
+ toDoc = toDV.nextDoc()) {
+ if (toLiveDocs != null && !toLiveDocs.get(toDoc)) {
+ continue;
+ }
+ for (int i = 0; i < toDV.docValueCount(); i++) {
+ toDocPlusOneByToOrd.set(toDV.nextOrd(), toDoc + 1L);
+ }
+ }
+ return toDocPlusOneByToOrd;
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+ }
+
+ /**
+ * Builds the join column for one (from-segment, to-segment) pair: resolves every from-side doc to
+ * its matching to-side doc id, along with the pair's from-doc and to-doc bounds. From-side terms
+ * are hashed by {@link ForeignKeyColumn}; each to-side term is looked up in that hash to map
+ * from-side ords to to-side ords.
+ *
+ * <p>Docs already deleted at build time are skipped, purely to avoid persisting entries nobody
+ * can ever match -- deletes are otherwise re-checked live at query time (from-side in {@code
+ * JoinIndexScorerSupplier}, to-side by the searcher's own {@code acceptDocs}), since a pair's
+ * cached mapping outlives whatever gets deleted after it was built.
+ */
+ static JoinColumnModel computeDocMapping(
+ LeafReaderContext toContext,
+ String toField,
+ ForeignKeyColumn fromSideData,
+ ToDocInvertor toDocInvertor)
+ throws IOException {
+ assert fromSideData != null;
+
+ SortedSetDocValues toDV = DocValues.getSortedSet(toContext.reader(), toField);
+ // addressed by from-ord in term order, not doc order, so this one stays mutable -- but paged
+ // and packed all the same: as a flat long[] it was 40MB for a five-million-term from side,
+ // allocated afresh for every pair built
+ PagedMutable toOrdPlusOneByFromOrd =
+ new PagedMutable(
+ fromSideData.getFromValuesCount(),
+ PAGE_SIZE,
+ PackedInts.bitsRequired(toDV.getValueCount()),
+ PackedInts.COMPACT);
+ TermsEnum toTerms = toDV.termsEnum();
+ // resolve from-side ords to to-side ords: look each to-side term up in the from-side hash.
+ boolean termsAreDisjoint = true;
+ for (BytesRef term = toTerms.next(); term != null; term = toTerms.next()) {
+ int fromOrd = fromSideData.getFromTermOrdOrDashOne(term);
+ if (fromOrd != -1) {
+ toOrdPlusOneByFromOrd.set(fromOrd, toTerms.ord() + 1);
+ termsAreDisjoint = false;
+ }
+ }
+ // TODO: this degrades M:N joins to M:1. Both toDocByToOrd and the resolved column keep a single
+ // to-side doc per from doc, so when several to docs share a term (non-unique toField) or a
+ // fromSideData doc is multi-valued with several matching terms, later assignments overwrite
+ // earlier ones and only the last match survives. The read side (AuxIndexJoinQuery) already
+ // consumes all docValueCount() values per doc, so only this writer needs to learn to emit
+ // multiple to docs per fromSideData doc.
+ if (termsAreDisjoint) { // no from-side term occurs on the to side: nothing to lay out at all
+ return JoinColumnModel.tombstone(fromSideData.fromSideMaxDocs());
+ }
+ PagedMutable toDocPlusOneByToOrd = toDocInvertor.apply(toContext);
+ int fromSideMaxDocs = fromSideData.fromSideMaxDocs();
+
+ // pass one: measure. Nothing is allocated until the match count is known, which is what decides
+ // the layout -- and lets the layout be allocated at exactly its final size.
+ int minFromDoc = DocIdSetIterator.NO_MORE_DOCS;
+ int maxFromDoc = -1;
+ int minToDoc = DocIdSetIterator.NO_MORE_DOCS;
+ int maxToDoc = -1;
+ int toCount = 0;
+ for (ForeignKeyColumn.FromOrds fromOrds = fromSideData.fromOrds(); fromOrds.next(); ) {
+ int toDoc = resolveToDoc(fromOrds.fromOrd(), toOrdPlusOneByFromOrd, toDocPlusOneByToOrd);
+ if (toDoc == -1) {
+ continue;
+ }
+ int fromDoc = fromOrds.fromDoc();
+ minFromDoc = Math.min(minFromDoc, fromDoc);
+ maxFromDoc = fromDoc;
+ minToDoc = Math.min(minToDoc, toDoc);
+ maxToDoc = Math.max(maxToDoc, toDoc);
+ toCount++;
+ }
+ if (maxFromDoc < 0) { // terms overlapped, but no live from-doc reaches a live to-doc
+ return JoinColumnModel.tombstone(fromSideMaxDocs);
+ }
+ DocEdges edges =
+ new Edges(new int[] {minFromDoc, maxFromDoc}, new int[] {minToDoc, maxToDoc}, toCount);
+
+ // pass two: lay the matches out, over the same walk. The dense layout lays down a slot per
+ // from-doc only as far as the last match, so that span, not the segment's width, is what it
+ // would cost -- and so what the layout is chosen on.
+ JoinColumnModel.Builder column =
+ JoinColumnModel.builder(
+ fromSideMaxDocs, JoinColumnModel.sparseIsSmaller(toCount, maxFromDoc + 1));
+ for (ForeignKeyColumn.FromOrds fromOrds = fromSideData.fromOrds(); fromOrds.next(); ) {
+ int toDoc = resolveToDoc(fromOrds.fromOrd(), toOrdPlusOneByFromOrd, toDocPlusOneByToOrd);
+ if (toDoc != -1) {
+ column.add(fromOrds.fromDoc(), toDoc);
+ }
+ }
+ assert column.matches() == toCount
+ : "measured " + toCount + " matches, laid out " + column.matches();
+ return column.build(edges);
+ }
+
+ /**
+ * The to-side doc a from-doc's {@code fromOrd} reaches, or {@code -1} when the from-doc has no
+ * value, its term has no to-side match, or that to-side term's only doc is deleted. Both build
+ * passes resolve every from-doc through here, so they agree on what counts as a match by
+ * construction. Both maps hold their values offset by one, so that the zero a paged map starts at
+ * reads back as "no value".
+ */
+ private static int resolveToDoc(
+ int fromOrd, PagedMutable toOrdPlusOneByFromOrd, PagedMutable toDocPlusOneByToOrd) {
+ if (fromOrd == -1) {
+ return -1;
+ }
+ long toOrdPlusOne = toOrdPlusOneByFromOrd.get(fromOrd);
+ return toOrdPlusOne == 0 ? -1 : (int) toDocPlusOneByToOrd.get(toOrdPlusOne - 1) - 1;
+ }
+
+ /**
+ * Reads a pair's persisted {@code {min, max}} edges (or {@code toCount}), all stored on doc 0 of
+ * the column -- the read-side counterpart of {@link JoinColumWriter}'s edges columns.
+ */
+ static int[] loadEdges(LeafReaderContext joinContext, String edgesFieldName) throws IOException {
+ SortedNumericDocValues edgesDV = joinContext.reader().getSortedNumericDocValues(edgesFieldName);
+ assert edgesDV != null : "expected edges column to be present: " + edgesFieldName;
+ int zeroDoc = edgesDV.nextDoc();
+ assert zeroDoc == 0 : "expected edges column to be fully materialized, but got doc " + zeroDoc;
+ int[] values = new int[edgesDV.docValueCount()];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (int) edgesDV.nextValue();
+ }
+ return values;
+ }
+
+ /**
+ * The join index field name addressing the ordinal map of one (from-segment, to-segment) pair.
+ */
+ static String pairFieldName(
+ LeafReaderContext fromContext,
+ String fromField,
+ LeafReaderContext toContext,
+ String toField) {
+ return getSideKey(fromContext, fromField) + "_" + getSideKey(toContext, toField);
+ }
+
+ /**
+ * Persistent identifier of one join side: the join field name, the immutable id the segment was
+ * created with (it survives reopens, growing deletes mask and reorderings of {@link
+ * IndexReader#leaves()}; a merge produces a new segment with a new id) and the docvalues
+ * generation of the join field.
+ */
+ /**
+ * Every side key {@code searcher}'s index currently offers for {@code field}: one per leaf, the
+ * same strings {@link #getSideKey} builds into a pair field name. A pair whose key is missing
+ * from this set names a segment that no longer exists, so its column can never be read again.
+ */
+ static Set<String> liveSideKeys(IndexSearcher searcher, String field) {
+ Set<String> keys = CollectionUtil.newHashSet(searcher.getIndexReader().leaves().size());
+ for (LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
+ keys.add(getSideKey(leaf, field));
+ }
+ return keys;
+ }
+
+ /**
+ * Splits a pair field name back into the two side keys {@link #pairFieldName} joined, or {@code
+ * null} when it isn't one.
+ *
+ * <p>The separator is found rather than searched for blindly: a side key is {@code
+ * field:segmentId:dvGen}, whose field name may itself contain underscores (PRODUCT_ID_FK) but no
+ * colon, and whose generation is a number. So the underscore that joins the two keys is the first
+ * one after the second colon -- everything between that colon and it is the generation.
+ */
+ static String[] splitPairFieldName(String pairFieldName) {
+ int fieldEnd = pairFieldName.indexOf(':');
+ if (fieldEnd < 0) {
+ return null;
+ }
+ int idEnd = pairFieldName.indexOf(':', fieldEnd + 1);
+ if (idEnd < 0) {
+ return null;
+ }
+ int separator = pairFieldName.indexOf('_', idEnd + 1);
+ if (separator < 0) {
+ return null;
+ }
+ return new String[] {
+ pairFieldName.substring(0, separator), pairFieldName.substring(separator + 1)
+ };
+ }
+
+ /** Whether {@code sideKey} is a key of {@code field}, as opposed to some other join field's. */
+ static boolean isKeyOf(String sideKey, String field) {
+ return sideKey.length() > field.length()
+ && sideKey.charAt(field.length()) == ':'
+ && sideKey.startsWith(field);
+ }
+
+ static String getSideKey(LeafReaderContext context, String field) {
+ byte[] segmentId = segmentReader(context.reader()).getSegmentInfo().info.getId();
+ // dvGen starts at -1 and advances only when this particular field receives an in-place
+ // IndexWriter.updateDocValues update; deletes only bump delGen and leave it untouched. So the
+ // key is insensitive to deletes but changes when the join field's docvalues are updated.
+ FieldInfo fieldInfo = context.reader().getFieldInfos().fieldInfo(field);
+ long dvGen = fieldInfo == null ? -1 : fieldInfo.getDocValuesGen();
+ String key = field + ":" + StringHelper.idToString(segmentId) + ":" + dvGen;
+ return key;
+ }
+
+ /**
+ * Peels wrappers off a leaf reader down to its {@link SegmentReader}. {@link
+ * FilterLeafReader#unwrap} alone is not enough: wrappers like {@code
+ * SoftDeletesDirectoryReaderWrapper} produce {@link FilterCodecReader} leaves, which are not
+ * {@link FilterLeafReader}s, and the two kinds may alternate. Tests additionally wrap readers in
+ * a {@link ParallelLeafReader} (e.g. {@code LuceneTestCase#newSearcher}); that class can in
+ * general combine several independent readers side by side, but when it wraps exactly one (the
+ * common case, including every reader the test framework wraps purely for coverage) that one is
+ * unambiguous and safe to descend into.
+ */
+ static SegmentReader segmentReader(LeafReader reader) {
+ while (true) {
+ if (reader instanceof SegmentReader segmentReader) {
+ return segmentReader;
+ } else if (reader instanceof FilterLeafReader filterLeafReader) {
+ reader = filterLeafReader.getDelegate();
+ } else if (reader instanceof FilterCodecReader filterCodecReader) {
+ reader = filterCodecReader.getDelegate();
+ } else if (reader instanceof ParallelLeafReader parallelLeafReader) {
+ LeafReader[] parallelReaders = parallelLeafReader.getParallelReaders();
+ if (parallelReaders.length != 1) {
+ throw new IllegalArgumentException(
+ "cannot unwrap a SegmentReader from a ParallelLeafReader combining "
+ + parallelReaders.length
+ + " independent readers");
+ }
+ reader = parallelReaders[0];
+ } else {
+ throw new IllegalArgumentException(
+ "cannot unwrap a SegmentReader from " + reader.getClass().getName());
+ }
+ }
+ }
+
+ /** The name of the sidecar segment carrying the given join index leaf. */
+ static String segmentName(LeafReaderContext joinContext) {
+ return segmentReader(joinContext.reader()).getSegmentName();
+ }
+
+ /**
+ * The {@link Directory} backing {@code reader}, unwrapped from any composite that lacks a single
+ * directory of its own (e.g. a {@link ParallelCompositeReader}): a {@link DirectoryReader}
+ * exposes its directory directly, anything else contributes the first leaf-reader whose directory
+ * owns it (descending through {@link ParallelLeafReader} and codec/filter wrappers).
+ */
+ static Directory directory(IndexReader reader) {
+ if (reader instanceof DirectoryReader dr) {
+ return dr.directory();
+ }
+ for (LeafReaderContext leaf : reader.leaves()) {
+ return segmentReader(leaf.reader()).getSegmentInfo().info.dir;
+ }
+ throw new IllegalArgumentException("no directory backing " + reader);
+ }
+
+ /**
+ * A hashable key identifying the storage location behind {@code directory}, stable across
+ * separate opens of the same path so repeated calls resolve to the same cache entry.
+ */
+ static Object directoryKey(Directory directory) throws IOException {
+ Directory unwrapped = FilterDirectory.unwrap(directory);
+ if (unwrapped instanceof FSDirectory fsDir) {
+ return fsDir.getDirectory().toRealPath(); // Path has proper equals/hashCode
+ }
+ return unwrapped; // RAMDirectory/ByteBuffersDirectory etc: identity is the real key
+ }
+
+ /**
+ * Every pair field name (the part after {@link #TO_COUNT_PREFIX}) present in {@code fieldInfos},
+ * i.e. every join pair this segment carries, needed or not. Detected by the toCount companion --
+ * written for every built pair -- not by the join column itself, which a tombstone pair (disjoint
+ * terms) never materializes; the reaper must see tombstones too, or it would judge a segment
+ * carrying only tombstones as pair-free and reap it, resurrecting the pairs' rebuilds.
+ */
+ static Set<String> pairFieldNames(FieldInfos fieldInfos) {
+ Set<String> names = new HashSet<>();
+ for (FieldInfo fieldInfo : fieldInfos) {
+ String[] splits = fieldInfo.name.split(TO_COUNT_PREFIX);
+ if (splits.length == 2) {
+ names.add(splits[1]);
+ }
+ }
+ return names;
+ }
+
+ /**
+ * Reads a segment's {@link FieldInfos} straight off disk, without opening a full reader. Mirrors
+ * {@code IndexWriter#readFieldInfos}, which isn't visible outside its package.
+ */
+ static FieldInfos readFieldInfos(SegmentCommitInfo info) throws IOException {
+ Codec codec = info.info.getCodec();
+ FieldInfosFormat fieldInfosFormat = codec.fieldInfosFormat();
+ if (info.hasFieldUpdates()) {
+ String segmentSuffix = Long.toString(info.getFieldInfosGen(), Character.MAX_RADIX);
+ return fieldInfosFormat.read(info.info.dir, info.info, segmentSuffix, IOContext.READONCE);
+ } else if (info.info.getUseCompoundFile()) {
+ try (Directory cfs = codec.compoundFormat().getCompoundReader(info.info.dir, info.info)) {
+ return fieldInfosFormat.read(cfs, info.info, "", IOContext.READONCE);
+ }
+ } else {
+ return fieldInfosFormat.read(info.info.dir, info.info, "", IOContext.READONCE);
+ }
+ }
+
+ // copy of org.apache.lucene.search.LRUQueryCache.cacheIntoRoaringDocIdSet
+ protected static class CacheAndCount implements Accountable {
+ protected static final CacheAndCount EMPTY = new CacheAndCount(DocIdSet.EMPTY, 0);
+
+ private static final long BASE_RAM_BYTES_USED =
+ RamUsageEstimator.shallowSizeOfInstance(CacheAndCount.class);
+ private final DocIdSet cache;
+ private final int count;
+
+ public CacheAndCount(DocIdSet cache, int count) {
+ this.cache = cache;
+ this.count = count;
+ }
+
+ public DocIdSetIterator iterator() throws IOException {
+ return cache.iterator();
+ }
+
+ // TODO is it really a cost or a count???
+ public int count() {
+ return count;
+ }
+
+ @Override
+ public long ramBytesUsed() {
+ return BASE_RAM_BYTES_USED + cache.ramBytesUsed();
+ }
+ }
+
+ static CacheAndCount cacheImpl(BulkScorer scorer, int maxDoc, Bits liveDocs) throws IOException {
+ if (scorer.cost() * 100 >= maxDoc) {
+ // FixedBitSet is faster for dense sets and will enable the random-access
+ // optimization in ConjunctionDISI
+ return cacheIntoBitSet(scorer, maxDoc, liveDocs);
+ } else {
+ return cacheIntoRoaringDocIdSet(scorer, maxDoc, liveDocs);
+ }
+ }
+
+ private static CacheAndCount cacheIntoBitSet(BulkScorer scorer, int maxDoc, Bits liveDocs)
+ throws IOException {
+ final FixedBitSet bitSet = new FixedBitSet(maxDoc);
+ int[] count = new int[1];
+ scorer.score(
+ new LeafCollector() {
+
+ private int[] buffer;
+
+ @Override
+ public void setScorer(Scorable scorer) throws IOException {}
+
+ @Override
+ public void collect(int doc) throws IOException {
+ if (liveDocs == null || liveDocs.get(doc)) {
+ count[0]++;
+ bitSet.set(doc);
+ }
+ }
+
+ @Override
+ public void collect(DocIdStream stream) throws IOException {
+ if (buffer == null) {
+ buffer = new int[128];
+ }
+ for (int c = stream.intoArray(buffer); c != 0; c = stream.intoArray(buffer)) {
+ int skip = 0;
+ for (int i = 0; i < c; ++i) {
+ if (liveDocs != null && !liveDocs.get(buffer[i])) {
+ skip++;
+ continue;
+ }
+ bitSet.set(buffer[i]);
+ }
+ count[0] += c - skip;
+ }
+ }
+ },
+ null,
+ 0,
+ DocIdSetIterator.NO_MORE_DOCS);
+ return new CacheAndCount(new BitDocIdSet(bitSet, count[0]), count[0]);
+ }
+
+ private static CacheAndCount cacheIntoRoaringDocIdSet(
+ BulkScorer scorer, int maxDoc, Bits liveDocs) throws IOException {
+ RoaringDocIdSet.Builder builder = new RoaringDocIdSet.Builder(maxDoc);
+ scorer.score(
+ new LeafCollector() {
+
+ private int[] buffer = null;
+
+ @Override
+ public void setScorer(Scorable scorer) throws IOException {}
+
+ @Override
+ public void collect(int doc) throws IOException {
+ if (liveDocs != null && !liveDocs.get(doc)) {
+ return;
+ }
+ builder.add(doc);
+ }
+
+ @Override
+ public void collect(DocIdStream stream) throws IOException {
+ if (buffer == null) {
+ buffer = new int[128];
+ }
+ for (int c = stream.intoArray(buffer); c != 0; c = stream.intoArray(buffer)) {
+ for (int i = 0; i < c; ++i) {
+ if (liveDocs != null && !liveDocs.get(buffer[i])) {
+ continue;
+ }
+ builder.add(buffer[i]);
+ }
+ }
+ }
+ },
+ null,
+ 0,
+ DocIdSetIterator.NO_MORE_DOCS);
+ RoaringDocIdSet cache = builder.build();
+ return new CacheAndCount(cache, cache.cardinality());
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexWeight.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexWeight.java
new file mode 100644
index 00000000000..da1ea8a8acd
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/JoinIndexWeight.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreMode;
+import org.apache.lucene.search.Scorer;
+import org.apache.lucene.search.ScorerSupplier;
+import org.apache.lucene.search.Weight;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager.JoinSegmentReference;
+
+/**
+ * @lucene.experimental
+ */
+final class JoinIndexWeight extends Weight {
+ final IndexSearcher maybeStaleJoinSearcher;
+ final Map<String, JoinSegmentReference> existingJoinSegments;
+ private final ScoreMode scoreMode;
+ private final float boost;
+ private final IndexReader toReader;
+ private final Map<Integer, Future<FromLeafJoinContext>> fromColumnFutures;
+
+ /**
+ * [toSegmentOrd][fromSegmentOrd] -> the pair column resolved at construction time; null where the
+ * from segment has no cached matches, the pair maps no from-side values, or no cached match falls
+ * into the pair's from-doc range. Pair columns record the sidecar segment name, not a leaf
+ * context, so they stay valid across join reader refreshes.
+ */
+ // private final PairColumn[][] pairColumnsByTo;
+
+ JoinIndexWeight(
+ AuxIndexJoinQuery auxIndexJoinQuery,
+ IndexSearcher maybeStaleJoinSearcher,
+ Map<String, JoinSegmentReference> existingJoinSegments,
+ IndexReader toReader,
+ ScoreMode scoreMode,
+ float boost,
+ Map<Integer, Future<FromLeafJoinContext>> foreignColsFutures) {
+ super(auxIndexJoinQuery);
+ this.maybeStaleJoinSearcher = maybeStaleJoinSearcher;
+ this.existingJoinSegments = existingJoinSegments;
+ this.scoreMode = scoreMode;
+ this.boost = boost;
+ this.toReader = toReader;
+ this.fromColumnFutures = foreignColsFutures;
+ }
+
+ private AuxIndexJoinQuery aiJQuery() {
+ return (AuxIndexJoinQuery) this.parentQuery;
+ }
+
+ @Override
+ public Explanation explain(LeafReaderContext context, int doc) throws IOException {
+ ScorerSupplier supplier = scorerSupplier(context);
+ if (supplier != null) {
+ Scorer scorer = supplier.get(1);
+ if (scorer.iterator().advance(doc) == doc) {
+ return Explanation.match(scorer.score(), aiJQuery().toString());
+ }
+ }
+ return Explanation.noMatch(aiJQuery().toString());
+ }
+
+ @Override
+ public int count(LeafReaderContext context) throws IOException {
+ // cost() is a range-size upper bound, not an exact match count, so counting has to
+ // fall back to actually driving the two-phase iterator
+ return super.count(context);
+ }
+
+ @Override
+ public ScorerSupplier scorerSupplier(LeafReaderContext tolrc) throws IOException {
+ IndexSearcher joinSearcher = aiJQuery().joinIndex.acquire();
+ try {
+ JoinIndexScorerSupplier ctx = null;
+ try {
+ ctx =
+ new JoinIndexScorerSupplier(
+ tolrc,
+ aiJQuery().fromField,
+ aiJQuery().fromQuery,
+ aiJQuery().fromSearcher,
+ aiJQuery().toField,
+ this.toReader,
+ this.existingJoinSegments,
+ this.maybeStaleJoinSearcher,
+ joinSearcher,
+ aiJQuery().joinIndex,
+ this.fromColumnFutures,
+ scoreMode,
+ boost);
+ } catch (ExecutionException e) { // TODO review exception
+ throw new RuntimeException(e);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return ctx.isEmpty() ? null : ctx;
+ } finally {
+ aiJQuery().joinIndex.release(joinSearcher);
+ }
+ }
+
+ @Override
+ public boolean isCacheable(LeafReaderContext lrc) {
+ // matches depend on the from-side searcher and the external join index, which the
+ // query cache cannot see
+ return false;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/package-info.java b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/package-info.java
new file mode 100644
index 00000000000..2afa1089800
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/search/join/auxindexjoin/package-info.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ *
+ *
+ * <h2>Auxiliary Index Join</h2>
+ *
+ * Persists doc-nums to doc-nums relation in auxiliary index.
+ *
+ * <h3>Simplified Collaboration Diagram</h3>
+ *
+ * <pre>
+ * {@link org.apache.solr.search.join.AuxIndexJoinQParserPlugin}
+ * │
+ * â–¼
+ * {@link org.apache.solr.search.join.auxindexjoin.AuxIndexManager} ──────▶ {@link org.apache.solr.search.join.auxindexjoin.AuxIndexJoinConfig}
+ * │ │
+ * │ └──────────▶ {@link org.apache.solr.search.join.auxindexjoin.AuxIndexJoinMergePolicy}
+ * â–¼
+ * {@link org.apache.solr.search.join.auxindexjoin.AuxIndexJoinQuery} ──────▶ {@link org.apache.solr.search.join.auxindexjoin.FromLeafJoinContext}
+ * │ │
+ * â–¼ â–¼
+ * {@link org.apache.solr.search.join.auxindexjoin.JoinIndexWeight} {@link org.apache.solr.search.join.auxindexjoin.ForeignKeyColumn}
+ * │
+ * â–¼
+ * {@link org.apache.solr.search.join.auxindexjoin.JoinIndexScorerSupplier} ──────▶ {@link org.apache.solr.search.join.auxindexjoin.JoinColumnIndexer}
+ * │ │ │ │
+ * │ ▼ │ ▼
+ * │{@link org.apache.solr.search.join.auxindexjoin.JoinIndexScorerSupplier.LeafJoin} │{@link org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel}
+ * â–¼ â–¼
+ * {@link org.apache.solr.search.join.auxindexjoin.JoinIndexScorerSupplier.LazyConfirmationIterator} {@link org.apache.solr.search.join.auxindexjoin.JoinColumnDocWriter}
+ *
+ * </pre>
+ *
+ * <p>{@link org.apache.solr.search.join.auxindexjoin.AuxIndexManager} owns an auxiliary Lucene
+ * index persisting, for every (from-segment, to-segment) pair, a SORTED_NUMERIC column mapping
+ * from-side doc ids to to-side doc ids, plus edges columns with the pair's {@code {min, max}} doc
+ * bounds and a to-side count. Pair columns are named by both sides' persistent side keys (segment
+ * id + docvalues generation of the join field), so they survive reopens of either side and are
+ * built lazily — the first {@code AuxIndexJoinQuery} weight that needs a pair writes it. See {@link
+ * org.apache.solr.search.join.auxindexjoin.AuxIndexManager} for the user-facing API and {@link
+ * org.apache.solr.search.join.auxindexjoin.AuxIndexJoinConfig} for tunables (blocking vs.
+ * non-blocking refresh, one-field-per-segment, the reaper's sweep interval).
+ *
+ * <p>A batch of pair columns is written through {@code JoinColumnDocWriter}. It guarantes a batch
+ * lands doc-for-doc in one sidecar segment, so doc 0's edges and every from-doc id line up the same
+ * way.
+ *
+ * <h2>Garbage collection of dead pairs</h2>
+ *
+ * The sidecar is append-only. A side key dies when its segment is merged away, dropped, or its join
+ * field receives an in-place docvalues update (dvGen bump); pair columns referencing a dead side
+ * key can never be read again. {@code AuxIndexJoinMergePolicy} reaps them:
+ *
+ * <ul>
+ * <li><b>Death signal via sampling, not listeners.</b> {@code AuxIndexManager#onCreateWeight}
+ * (called from {@code AuxIndexJoinQuery#createWeight}) reports the set of pair field names a
+ * query actually needed to {@code AuxIndexJoinMergePolicy#onCreateWeight}, keyed by
+ * (from-directory, to-directory). A pair field name present in an earlier snapshot for the
+ * same searcher pair but missing from a later one is queued as a pending removal. Sampling is
+ * throttled ({@code AuxIndexJoinConfig#setSweepSamplingInterval}, default one minute) since
+ * it is only a heuristic hint, not a correctness requirement, and both the snapshot map and
+ * the pending-removal set are size-bounded (best-effort LRU-ish eviction).
+ * <li><b>Only a newer view may condemn.</b> The snapshot is keyed by directory, which is stable
+ * across reopens, so consecutive samples come from different queries that may hold different
+ * searcher generations. A sample taken from an older generation than the stored snapshot is
+ * discarded rather than acted on: it cannot see the segments opened since, and would report
+ * every one of them as dead. Readers with no orderable version are trusted as before.
+ * <li><b>Reaping.</b> {@code findMerges} reclaims dead pairs two ways, both piggybacked on
+ * ordinary merges — there is no background thread. A segment whose pair field names are
+ * <em>all</em> pending removals is dropped whole, via a {@code OneMerge} that reports it
+ * fully deleted ({@code wrapForMerge} returns a {@code MatchNoBits} live-docs view) so {@code
+ * IndexWriter} discards it instead of rewriting it. A segment only <em>partly</em> dead — the
+ * usual case once a segment carries hundreds of pairs — is rewritten by a {@code
+ * DocAlignedMerge} with the dead columns hidden from its {@code FieldInfos}, either folded
+ * into a compaction it was going to take part in anyway or, failing that, as a merge of one.
+ * Either way the names come off the queue only once the merge commits. {@code
+ * TestAuxIndexJoinMergePolicy} and {@code droppedSegmentCount()} / {@code
+ * purgedSegmentCount()} / {@code reapedPairCount()} cover this end to end.
+ * <li><b>Recoverable, not authoritative.</b> Reaping is a heuristic, so a query may still hold a
+ * reference to a column it drops. Such a pair is rebuilt by {@code
+ * JoinIndexScorerSupplier#refreshJoinTasksReferences} — including reading the from-side
+ * foreign-key column the weight skipped, precisely because the pair existed back then — and
+ * the cell is rebound to the rebuilt model mid-flight, which is safe because a cell's
+ * iteration state lives in its from-side iterator and never in the column. The {@code
+ * rebindsAfterReap} counter on the {@code evt=done} line reports how often this happens; it
+ * should be zero in a steady run.
+ * <li><b>Death by evidence, for what sampling cannot see.</b> The snapshot diff can only condemn
+ * a pair that one query needed and a later query didn't, both inside one process. Everything
+ * a restart inherits therefore lives forever: nobody needs a column whose from-segment was
+ * merged away while Solr was down, so nobody misses it either, and load tests duly ended with
+ * most of the sidecar's bytes in such columns. So the same sample also asks the readers
+ * directly: {@code JoinIndexUtils#liveSideKeys} enumerates the side keys the from and to
+ * indexes currently offer for the query's two fields, and any stored pair naming one they
+ * don't is queued outright. Attribution is by field name, and the from half is judged only
+ * while a single from-index has been seen joining on that name; the to side needs no such
+ * guard, the sidecar living under the to-core's dataDir. Counted by {@code
+ * strandedColumnCount()}.
+ * <li><b>Known gaps.</b> The sweep above sees only the field pairs that are actually queried:
+ * columns of a join nobody runs any more are still stranded. Sampling remains throttled, so
+ * reclaim lands within a sampling interval of the first commit, not immediately.
+ * </ul>
+ *
+ * <h2>Compaction of the sidecar</h2>
+ *
+ * The sidecar gains a segment per written batch and, unlike an ordinary index, cannot merge them
+ * the ordinary way: a sidecar doc id <em>is</em> the from-side doc id its column is addressed by,
+ * so concatenating two segments would shift every mapping in the second one. Left alone the segment
+ * count therefore grew without bound under a steady stream of pair builds, until the JVM ran out of
+ * mmap-able address space. {@code AuxIndexJoinMergePolicy} folds groups of them into one with a
+ * {@code DocAlignedMerge} instead, which unions their columns doc-for-doc -- doc {@code i} of the
+ * result carries doc {@code i} of every input, and the result is as long as its longest input, not
+ * as long as all of them together. Pair columns are opaque to it: their names already carry both
+ * sides' segment ids, so no two inputs' columns can collide.
+ *
+ * <p>Compaction bounds the segment count; what bounds the bytes is purging dead columns out of the
+ * segments that hold them, and that decision is made in bytes rather than in share of columns. A
+ * share reads well -- one dead column in ten pays for the rewrite -- but ranks this index exactly
+ * wrong: dead columns land in whatever segment compaction last folded them into, so the widest
+ * segments dilute their own dead share below any fixed percentage and become the least likely to be
+ * cleaned, though they are where the bytes are. Ranking by {@code reclaimableBytes} instead puts
+ * the fat segments first and leaves the thin ones queued until more of them dies; see {@code
+ * AuxIndexJoinConfig#setMinReclaimableBytesToPurge}.
+ *
+ * <h2>TODO</h2>
+ *
+ * <ul>
+ * <li>comparative benchmarking for updates: parent, children field, PARENT_ID_FK ({@code
+ * AIJoinBenchmark} currently only compares against {@code JoinUtil} on a static index)
+ * <li>many-to-many: {@code JoinIndexUtils} doc mapping currently degrades M:N joins to M:1
+ * (single to-doc kept per from-doc)
+ * <li>reverse join on the same columns: children by parents filter
+ * <li>we estimate to&from set by a range. Alternatives: union of ranges, roaring (bitset).
+ * Note: to and from estimates might be built different. For "to" side we need to provide
+ * advance()-eble iterator over union. And for "from" side it should be just intersectable
+ * with docSetIter. Format should balance storage size and decoding efforts. It should be
+ * stored as a Document fields.
+ * <li>let {@code DocAlignedMerge} drop the columns of pairs already queued for reaping while it
+ * rewrites a segment anyway: nearly free there, and the only way to reclaim a dead pair that
+ * shares a segment with live ones. Needs {@code
+ * JoinIndexScorerSupplier#refreshJoinTasksReferences} to rebuild a pair that vanished under a
+ * live query first, where it currently throws.
+ * <li>stripe columns for join index: break {@code JoinIndexUtils.TO_DOC_VAL_BY_FROM_DOCNUM} to a
+ * pair one is {@code to_doc_nums<maxdocs(to-side)/2} and {@code
+ * to_doc_nums>=maxdocs(to-side)/2}
+ * </ul>
+ *
+ * @lucene.experimental
+ * <p>This is experimental API and subject to change.
+ */
+package org.apache.solr.search.join.auxindexjoin;
diff --git a/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/schema.xml b/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/schema.xml
new file mode 100644
index 00000000000..f465bb48d5f
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/schema.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<!-- version 1.7: docValues default to true, which AuxIndexManager requires: it reads real
+ per-segment SortedSetDocValues directly, unlike JoinUtil-based {!join} which also tolerates
+ uninverted fields. -->
+<schema name="auxIndexJoin" version="1.7">
+ <fieldType name="string" class="solr.StrField"/>
+ <fieldType name="plong" class="solr.LongPointField"/>
+
+ <field name="id" type="string" required="true"/>
+ <field name="_version_" type="plong" indexed="true" stored="true"/>
+
+ <dynamicField name="*_s" type="string"/>
+
+ <uniqueKey>id</uniqueKey>
+</schema>
diff --git a/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/solrconfig.xml
new file mode 100644
index 00000000000..d30c890c7b2
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/auxindexjoin/conf/solrconfig.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<config>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <requestHandler name="/select" class="solr.SearchHandler" />
+
+ <queryParser name="auxIndexJoin" class="org.apache.solr.search.join.AuxIndexJoinQParserPlugin"/>
+
+</config>
diff --git a/solr/core/src/test/org/apache/solr/cloud/DistribAuxIndexJoinFromCollectionTest.java b/solr/core/src/test/org/apache/solr/cloud/DistribAuxIndexJoinFromCollectionTest.java
new file mode 100644
index 00000000000..0f3f49ef6eb
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/cloud/DistribAuxIndexJoinFromCollectionTest.java
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import static org.apache.solr.cloud.MiniSolrCloudCluster.DEFAULT_CLOUD_SOLR_XML;
+
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.solr.client.solrj.RemoteSolrException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.search.join.AuxIndexJoinQParserPlugin;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link AuxIndexJoinQParserPlugin} version of {@link DistribJoinFromCollectionTest}: same
+ * collocated-replica {@code fromIndex} setup (a single-shard "from" collection deployed onto every
+ * node holding a "to" replica, so {@link
+ * org.apache.solr.search.join.ScoreJoinQParserPlugin#getCoreName}'s collocation lookup always finds
+ * a local "from" replica), but adapted for two ways {@code {!auxIndexJoin}} differs from {@code
+ * {!join}}:
+ *
+ * <ul>
+ * <li>no scoring -- {@code {!auxIndexJoin}} is always a constant-score match, so there's no
+ * {@code score=} local param and no scoring assertions.
+ * <li>M:1 only -- {@code JoinIndexUtil#computeDocMapping} keeps exactly one to-doc per from-doc,
+ * so the join here always goes from the "from" collection's docs (each with a single-valued
+ * FK) to the "to" collection's docs by their unique key, never the reverse (one "to" doc
+ * resolving to many "from" docs would silently drop matches).
+ * </ul>
+ */
+public class DistribAuxIndexJoinFromCollectionTest extends SolrCloudTestCase {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String toColl = "aijoin_to_2x2";
+ private static final String fromColl = "aijoin_from_1x4";
+
+ private static String toDocId;
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ String configName = "aijoinCloudCollectionConfig";
+ int nodeCount = 5;
+ configureCluster(nodeCount)
+ .withSolrXml(
+ DEFAULT_CLOUD_SOLR_XML.replace(
+ "<solr>", "<solr><int name=\"indexSearcherExecutorThreads\">4</int>"))
+ .addConfig(configName, configset("auxindexjoin"))
+ .configure();
+
+ Map<String, String> collectionProperties = new HashMap<>();
+
+ // create a collection holding data for the "to" (unique-key) side of the JOIN
+ int shards = 2;
+ int replicas = 2;
+ CollectionAdminRequest.createCollection(toColl, configName, shards, replicas)
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
+ // get the set of nodes where replicas for the "to" collection exist
+ Set<String> nodeSet = new HashSet<>();
+ ZkStateReader zkStateReader = cluster.getZkStateReader();
+ ClusterState cs = zkStateReader.getClusterState();
+ for (Slice slice : cs.getCollection(toColl).getActiveSlices())
+ for (Replica replica : slice.getReplicas()) nodeSet.add(replica.getNodeName());
+ assertTrue(nodeSet.size() > 0);
+
+ // deploy the single-shard "from" collection to every node where the "to" collection exists,
+ // so AuxIndexJoinQParserPlugin's collocation lookup always finds a local "from" replica -- and
+ // stays
+ // on the single-shard fast path, which skips the router-field cross-checks that a sharded
+ // "from" collection would otherwise require
+ CollectionAdminRequest.createCollection(fromColl, configName, 1, 4)
+ .setCreateNodeSet(String.join(",", nodeSet))
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
+ toDocId = indexToDoc(1001, "b");
+ indexFromDoc(2001, toDocId, "c");
+ indexFromDoc(2002, toDocId, "d"); // a second "from" doc mapping to the same "to" doc: still M:1
+
+ Thread.sleep(1000); // so the commits fire
+ }
+
+ @AfterClass
+ public static void shutdown() {
+ log.info(
+ "DistribAuxIndexJoinFromCollectionTest logic complete ... deleting the {} and {} collections",
+ toColl,
+ fromColl);
+
+ // try to clean up
+ for (String c : new String[] {toColl, fromColl}) {
+ try {
+ CollectionAdminRequest.Delete req = CollectionAdminRequest.deleteCollection(c);
+ req.process(cluster.getSolrClient());
+ } catch (Exception e) {
+ // don't fail the test
+ log.warn("Could not delete collection {} after test completed due to:", c, e);
+ }
+ }
+
+ log.info("DistribAuxIndexJoinFromCollectionTest succeeded ... shutting down now!");
+ }
+
+ @Test
+ public void testJoin() throws Exception {
+ // verify the join with fromIndex works: two different "from" docs (match_s:c and match_s:d)
+ // both resolve to the same single "to" doc, still M:1 since it's one to-doc per from-doc
+ CloudSolrClient client = cluster.getSolrClient();
+ {
+ final String joinQ = "{!auxIndexJoin from=join_s fromIndex=" + fromColl + " to=id}match_s:c";
+ QueryRequest qr =
+ new QueryRequest(params("collection", toColl, "q", joinQ, "fl", "id,get_s,score"));
+ QueryResponse rsp = qr.process(client);
+ SolrDocumentList hits = rsp.getResults();
+ assertEquals("Expected 1 doc, got " + hits, 1, hits.getNumFound());
+ SolrDocument doc = hits.get(0);
+ assertEquals(toDocId, doc.getFirstValue("id"));
+ assertEquals("b", doc.getFirstValue("get_s"));
+ // {!auxIndexJoin} never scores -- always a constant-score match, like {!join score=none}
+ assertEquals("1.0", doc.getFirstValue("score").toString());
+ }
+
+ // negative test before creating an alias
+ checkAbsentFromIndex();
+
+ // create an alias for the fromIndex and then query through the alias
+ String alias = fromColl + "Alias";
+ CollectionAdminRequest.createAlias(alias, fromColl).process(client);
+
+ {
+ final String joinQ = "{!auxIndexJoin from=join_s fromIndex=" + alias + " to=id}match_s:d";
+ final QueryRequest qr =
+ new QueryRequest(params("collection", toColl, "q", joinQ, "fl", "id,get_s,score"));
+ final QueryResponse rsp = qr.process(client);
+ final SolrDocumentList hits = rsp.getResults();
+ assertEquals("Expected 1 doc", 1, hits.getNumFound());
+ SolrDocument doc = hits.get(0);
+ assertEquals(toDocId, doc.getFirstValue("id"));
+ assertEquals("b", doc.getFirstValue("get_s"));
+ }
+
+ // negative test after creating an alias
+ checkAbsentFromIndex();
+
+ {
+ // verify join doesn't work if no match in the "from" index
+ final String joinQ =
+ "{!auxIndexJoin from=join_s fromIndex=" + fromColl + " to=id}match_s:nomatch";
+ final QueryRequest qr =
+ new QueryRequest(params("collection", toColl, "q", joinQ, "fl", "id,get_s,score"));
+ final QueryResponse rsp = qr.process(client);
+ final SolrDocumentList hits = rsp.getResults();
+ assertEquals("Expected no hits", 0, hits.getNumFound());
+ }
+ }
+
+ private void checkAbsentFromIndex() {
+ final String wrongName = fromColl + "WrongName";
+ final String joinQ = "{!auxIndexJoin from=join_s fromIndex=" + wrongName + " to=id}match_s:c";
+ final QueryRequest qr =
+ new QueryRequest(params("collection", toColl, "q", joinQ, "fl", "id,get_s,score"));
+ RemoteSolrException ex =
+ assertThrows(RemoteSolrException.class, () -> cluster.getSolrClient().request(qr));
+ assertEquals(SolrException.ErrorCode.BAD_REQUEST.code, ex.code());
+ assertTrue(ex.getMessage().contains(wrongName));
+ }
+
+ /** Indexes a "to"-side doc: {@code id} is the join target, unique per doc (required for M:1). */
+ private static String indexToDoc(int id, String getField) throws Exception {
+ UpdateRequest up = new UpdateRequest();
+ up.setCommitWithin(50);
+ up.setParam("collection", toColl);
+ SolrInputDocument doc = new SolrInputDocument();
+ String docId = "" + id;
+ doc.addField("id", docId);
+ doc.addField("get_s", getField);
+ up.add(doc);
+ cluster.getSolrClient().request(up);
+ return docId;
+ }
+
+ /** Indexes a "from"-side doc with a single-valued {@code join_s} FK pointing to a "to" doc. */
+ private static void indexFromDoc(int id, String joinFieldValue, String matchField)
+ throws Exception {
+ UpdateRequest up = new UpdateRequest();
+ up.setCommitWithin(50);
+ up.setParam("collection", fromColl);
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", "" + id);
+ doc.addField("join_s", joinFieldValue);
+ doc.addField("match_s", matchField);
+ up.add(doc);
+ cluster.getSolrClient().request(up);
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/TestAuxIndexJoinQParserPlugin.java b/solr/core/src/test/org/apache/solr/search/join/TestAuxIndexJoinQParserPlugin.java
new file mode 100644
index 00000000000..4c1871f8d36
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/TestAuxIndexJoinQParserPlugin.java
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join;
+
+import static org.apache.solr.SolrTestCaseJ4.DEFAULT_TEST_COLLECTION_NAME;
+import static org.apache.solr.SolrTestCaseJ4.assertFieldValues;
+import static org.apache.solr.SolrTestCaseJ4.configset;
+import static org.apache.solr.SolrTestCaseJ4.params;
+
+import java.nio.file.Path;
+import java.util.List;
+import org.apache.commons.io.file.PathUtils;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.cloud.DistribAuxIndexJoinFromCollectionTest;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.search.join.auxindexjoin.AuxIndexManager;
+import org.apache.solr.util.EmbeddedSolrServerTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * Exercises {@link AuxIndexJoinQParserPlugin}, modeled on {@link TestScoreJoinQPNoScore}'s
+ * same-core {@code {!join}} coverage plus {@link org.apache.solr.TestCrossCoreJoin}'s cross-core
+ * setup, since cross-core joins -- not same-core -- are the reason {@link AuxIndexManager} exists.
+ *
+ * <p>Every join here goes from the "many" side (an employee, single-valued FK) to the "few" side
+ * (that employee's department, a value unique per to-doc): {@code AIJoinUtil#computeDocMapping} is,
+ * per its own javadoc, "always single-valued until M:N pairs are supported" -- it keeps exactly one
+ * to-doc per from-doc. That's exact for this direction (each employee has exactly one department),
+ * but would silently drop matches for the reverse, one-department-to-many-employees direction, so
+ * this test doesn't exercise that one.
+ *
+ * <p>Both cores use version 1.7 schemas ({@code configsets/aijoin}, shared with {@link
+ * DistribAuxIndexJoinFromCollectionTest}, and {@code configsets/minimal}), where docValues default
+ * to true: {@link AuxIndexManager} reads real per-segment {@link
+ * org.apache.lucene.index.SortedSetDocValues} directly, unlike {@link ScoreJoinQParserPlugin} (via
+ * {@link org.apache.lucene.search.join.JoinUtil}) which tolerates uninverted fields too.
+ */
+public class TestAuxIndexJoinQParserPlugin extends SolrTestCase {
+
+ private static final String FROM_CORE = "aijoinFromCore";
+
+ @ClassRule
+ public static final EmbeddedSolrServerTestRule solrRule = new EmbeddedSolrServerTestRule();
+
+ @BeforeClass
+ public static void beforeTests() throws Exception {
+ solrRule.startSolr();
+
+ // the aijoin configset declares no schemaFactory, so loading it file-based would let the
+ // default managed-schema factory rewrite the source-tree conf dir; work on a temp copy
+ Path aijoinConfigSet = createTempDir("auxindexjoin");
+ PathUtils.copyDirectory(configset("auxindexjoin"), aijoinConfigSet.resolve("conf"));
+ solrRule.newCollection().withConfigSet(aijoinConfigSet).create();
+
+ // "minimal" uses ClassicIndexSchemaFactory, which never writes, so it's safe to use in place
+ solrRule.newCollection(FROM_CORE).withConfigSet(configset("minimal")).create();
+
+ SolrClient toSide = solrRule.getSolrClient();
+
+ // departments: the "to"/unique side of every join below, one doc per dept_id_s value
+ toSide.add(
+ List.of(
+ doc("id", "10", "dept_id_s", "Engineering", "text_s", "These guys develop stuff"),
+ doc("id", "11", "dept_id_s", "Marketing", "text_s", "These guys make you look good"),
+ doc("id", "12", "dept_id_s", "Sales", "text_s", "These guys sell stuff"),
+ doc("id", "13", "dept_id_s", "Support", "text_s", "These guys help customers")));
+
+ // employees: the "from"/many side, each with exactly one dept_s value
+ toSide.add(employees());
+ toSide.commit();
+
+ // aijoinFromCore holds the same employees, for the cross-core variant of the same joins
+ SolrClient fromSide = solrRule.getSolrClient(FROM_CORE);
+ fromSide.add(employees());
+ fromSide.commit();
+ }
+
+ private static List<SolrInputDocument> employees() {
+ return List.of(
+ doc("id", "1", "name_s", "john", "title_s", "Director", "dept_s", "Engineering"),
+ doc("id", "2", "name_s", "mark", "title_s", "VP", "dept_s", "Marketing"),
+ doc("id", "3", "name_s", "nancy", "title_s", "MTS", "dept_s", "Sales"),
+ doc("id", "4", "name_s", "dave", "title_s", "MTS", "dept_s", "Support"),
+ doc("id", "5", "name_s", "tina", "title_s", "VP", "dept_s", "Engineering"));
+ }
+
+ private static SolrInputDocument doc(String... fieldsAndValues) {
+ return new SolrInputDocument(fieldsAndValues);
+ }
+
+ /** Queries the main core and asserts the matches' ids, in index (docid) order. */
+ private static void assertJoin(SolrParams query, String... expectedIds) throws Exception {
+ SolrDocumentList results = solrRule.getSolrClient().query(query).getResults();
+ assertEquals(results.toString(), expectedIds.length, results.getNumFound());
+ assertFieldValues(results, "id", (Object[]) expectedIds);
+ }
+
+ @Test
+ public void testMissingLocalParams() {
+ SolrException e =
+ expectThrows(
+ SolrException.class,
+ () -> solrRule.getSolrClient().query(params("q", "{!auxIndexJoin to=dept_id_s}*:*")));
+ assertEquals(SolrException.ErrorCode.BAD_REQUEST.code, e.code());
+ assertTrue(e.getMessage(), e.getMessage().contains("requires"));
+ }
+
+ @Test
+ public void testSameCoreJoin() throws Exception {
+ // nancy and dave (title MTS) each belong to exactly one dept: Sales and Support
+ assertJoin(
+ params("q", "{!auxIndexJoin from=dept_s to=dept_id_s}title_s:MTS", "fl", "id"), "12", "13");
+
+ // from-side subordinate query matches nothing
+ assertJoin(
+ params("q", "{!auxIndexJoin from=dept_s to=dept_id_s}name_s:nosuchperson", "fl", "id"));
+
+ // to-side field has real values, but none equal to any matched from-side value
+ // ("Engineering" is a dept_s value, but no title_s is ever "Engineering")
+ assertJoin(params("q", "{!auxIndexJoin from=dept_s to=title_s}name_s:john", "fl", "id"));
+
+ // a single from-doc resolves to exactly its one department
+ assertJoin(
+ params("q", "{!auxIndexJoin from=dept_s to=dept_id_s}name_s:john", "fl", "id"), "10");
+
+ // variable deref for sub-query parsing, plus defType local param
+ assertJoin(
+ params(
+ "q", "{!auxIndexJoin from=dept_s to=dept_id_s v=$qq}",
+ "qq", "{!dismax qf=name_s}dave",
+ "fl", "id"),
+ "13");
+
+ // fq on the to-side (department) is pushed down alongside the join, mirroring SOLR-3062 for
+ // {!join}: john and tina (title VP) join to Engineering and Marketing, fq narrows to just one
+ assertJoin(
+ params(
+ "q", "{!auxIndexJoin from=dept_s to=dept_id_s}title_s:VP",
+ "fl", "id",
+ "fq", "dept_id_s:Engineering"),
+ "10");
+ }
+
+ @Test
+ public void testCrossCoreJoin() throws Exception {
+ // nancy and dave (title MTS) live in aijoinFromCore; their departments live in this core
+ assertJoin(
+ params(
+ "q",
+ "{!auxIndexJoin from=dept_s to=dept_id_s fromIndex=" + FROM_CORE + "}title_s:MTS",
+ "fl",
+ "id"),
+ "12",
+ "13");
+
+ // fq on the querying (to) core is still pushed down alongside the cross-core join
+ assertJoin(
+ params(
+ "q",
+ "{!auxIndexJoin from=dept_s to=dept_id_s fromIndex=" + FROM_CORE + "}title_s:VP",
+ "fl",
+ "id",
+ "fq",
+ "dept_id_s:Engineering"),
+ "10");
+
+ // the cross-core join must not leak a close on either core
+ CoreContainer coreContainer = solrRule.getCoreContainer();
+ try (SolrCore fromCore = coreContainer.getCore(FROM_CORE);
+ SolrCore toCore = coreContainer.getCore(DEFAULT_TEST_COLLECTION_NAME)) {
+ assertFalse(fromCore.isClosed());
+ assertFalse(toCore.isClosed());
+ }
+ }
+
+ /**
+ * A core reload informs the new core's plugin instance while the old core still holds the
+ * sidecar's {@code write.lock}, so the reloaded core has to join the already open {@link
+ * AuxIndexManager} rather than open a second one on the same directory.
+ */
+ @Test
+ public void testCoreReload() throws Exception {
+ SolrParams join =
+ params("q", "{!auxIndexJoin from=dept_s to=dept_id_s}title_s:MTS", "fl", "id");
+ assertJoin(join, "12", "13"); // builds pair columns, so the sidecar has state to carry over
+
+ CoreContainer coreContainer = solrRule.getCoreContainer();
+ AuxIndexManager sidecar = sidecarOf(coreContainer);
+ coreContainer.reload(DEFAULT_TEST_COLLECTION_NAME);
+
+ assertSame("the reloaded core must join the open sidecar", sidecar, sidecarOf(coreContainer));
+ assertJoin(join, "12", "13");
+ assertJoin(
+ params(
+ "q",
+ "{!auxIndexJoin from=dept_s to=dept_id_s fromIndex=" + FROM_CORE + "}title_s:MTS",
+ "fl",
+ "id"),
+ "12",
+ "13");
+ }
+
+ private static AuxIndexManager sidecarOf(CoreContainer coreContainer) {
+ try (SolrCore core = coreContainer.getCore(DEFAULT_TEST_COLLECTION_NAME)) {
+ AuxIndexJoinQParserPlugin plugin =
+ (AuxIndexJoinQParserPlugin) core.getQueryPlugin(AuxIndexJoinQParserPlugin.NAME);
+ return plugin.getJoinIndex();
+ }
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinBenchmark.java b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinBenchmark.java
new file mode 100644
index 00000000000..7b97d499ac9
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/AuxIndexJoinBenchmark.java
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.Locale;
+import java.util.concurrent.ExecutorService;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.SortedDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.NoMergePolicy;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.TopScoreDocCollectorManager;
+import org.apache.lucene.search.join.JoinUtil;
+import org.apache.lucene.search.join.ScoreMode;
+import org.apache.lucene.store.ByteBuffersDirectory;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+
+/**
+ * Wall-clock comparison of {@link JoinUtil} (the term-based variant, no global ordinals) against
+ * {@link AuxIndexJoinQuery} on a synthetic M:1 join: ~100K children in a few segments point to ~10K
+ * unique parents in a few segments through single valued sorted string docvalues. A small
+ * child-side term filter selects ~10K children that join to ~1K parents; a second case adds a
+ * parent-side filter on top of the join. Results are not asserted, only timed: each search repeats
+ * {@link #PASSES} times and min/max/avg wall times are printed.
+ *
+ * <p>The whole comparison runs {@link #ROUNDS} times over a growing, mutating index: between rounds
+ * the parent id boundary is raised by 10% and a tenth of the (new) parent population, with all
+ * their children, is rewritten through {@code updateDocument} at random ids below the new boundary
+ * — ids above the old boundary append fresh docs while ids below it clash with existing ones and
+ * replace them, leaving deletes behind. The {@link AuxIndexManager} is opened once and reused by
+ * every round and pass: after each update the first search builds only the missing pair columns,
+ * which is timed separately from the steady-state passes.
+ *
+ * <p>Plain Java benchmark, run directly with {@code main}, no test framework involved.
+ */
+public class AuxIndexJoinBenchmark {
+
+ private static final String PARENT_ID = "parent_id";
+ private static final String PARENT_ID_FK = "parent_id_FK";
+ private static final String CHILD_ID = "child_id";
+ private static final String TAG = "tag";
+ private static final String COLOR = "color";
+
+ private static final String HOT = "hot";
+ private static final String COLD = "cold";
+ // the parent filter matches one of COLOR_CARDINALITY colors, passing ~1/30 of the joined parents
+ private static final int COLOR_CARDINALITY = 30;
+
+ private static final int NUM_PARENTS = 100_000;
+ private static final int CHILDREN_PER_PARENT = 10;
+ // every HOT_STRIDE'th parent is "hot" and all its children carry the hot tag: the child filter
+ // TermQuery(tag:hot) matches NUM_PARENTS / HOT_STRIDE * CHILDREN_PER_PARENT ~ 10K children
+ // joining to NUM_PARENTS / HOT_STRIDE ~ 1K parents, scattered over all segments
+ private static final int HOT_STRIDE = 10;
+ private static final int PARENT_SEGMENTS = 4;
+ private static final int CHILD_SEGMENTS = 5;
+ private static final int PASSES = 10;
+ // benchmark rounds: between rounds the parent boundary grows by GROWTH_DENOMINATOR'th and
+ // 1/UPDATE_DENOMINATOR of the grown population is rewritten at random ids below the new boundary
+ private static final int ROUNDS = 10;
+ private static final int GROWTH_DENOMINATOR = 10;
+ private static final int UPDATE_DENOMINATOR = 10;
+
+ private long random = System.nanoTime(); // forbiddenApi pushed Random() usage
+
+ private interface SearchTask {
+ TopDocs run() throws IOException;
+ }
+
+ public static void main(String[] args) throws Exception {
+ new AuxIndexJoinBenchmark().runBenchmark();
+ // executor.shutdown();
+ }
+
+ public void runBenchmark() throws Exception {
+ // the auxiliary join index is opened once and reused by every round and pass
+ try (Directory parentsDir = new ByteBuffersDirectory();
+ Directory childrenDir = new ByteBuffersDirectory();
+ Directory joinDir = new ByteBuffersDirectory();
+ ExecutorService executor =
+ ExecutorUtil.newMDCAwareFixedThreadPool(
+ Runtime.getRuntime().availableProcessors(),
+ new SolrNamedThreadFactory(MethodHandles.lookup().lookupClass().getSimpleName()));
+ AuxIndexManager joinIndex = new AuxIndexManager(joinDir)) {
+ buildIndices(parentsDir, childrenDir);
+ int numParents = NUM_PARENTS;
+ for (int round = 0; round < ROUNDS; round++) {
+ benchmarkRound(round, numParents, parentsDir, childrenDir, joinIndex, executor);
+ if (round < ROUNDS - 1) {
+ numParents += numParents / GROWTH_DENOMINATOR;
+ updateIndices(parentsDir, childrenDir, numParents);
+ }
+ }
+ System.out.println("start warming");
+ try (IndexReader parentsReader = DirectoryReader.open(parentsDir);
+ IndexReader childrenReader = DirectoryReader.open(childrenDir)) {
+ IndexSearcher childrenSearcher = new ParallelIndexSearcher(childrenReader, executor);
+ childrenSearcher.setQueryCache(null);
+ IndexSearcher parentsSearcher = new ParallelIndexSearcher(parentsReader, executor);
+ parentsSearcher.setQueryCache(null);
+
+ Query childFilter = new TermQuery(new Term(TAG, HOT));
+ Query parentFilter = new TermQuery(new Term(COLOR, color(0)));
+
+ for (int i = 0; i < 10000; i++) {
+ exactSearch(
+ parentsSearcher,
+ filterParents(
+ aiJoinChildrenToParents(joinIndex, childFilter, childrenSearcher, executor),
+ parentFilter));
+ if (i % 100 == 0 && i > 0) {
+ System.out.println(i);
+ }
+ }
+ }
+ }
+ }
+
+ private void benchmarkRound(
+ int round,
+ int numParents,
+ Directory parentsDir,
+ Directory childrenDir,
+ AuxIndexManager joinIndex,
+ ExecutorService executor)
+ throws IOException {
+ try (IndexReader parentsReader = DirectoryReader.open(parentsDir);
+ IndexReader childrenReader = DirectoryReader.open(childrenDir)) {
+ System.out.printf(
+ Locale.ROOT,
+ "round %d: parent ids < %d, parents: %d/%d docs (live/max) / %d segments,"
+ + " children: %d/%d docs (live/max) / %d segments%n",
+ round,
+ numParents,
+ parentsReader.numDocs(),
+ parentsReader.maxDoc(),
+ parentsReader.leaves().size(),
+ childrenReader.numDocs(),
+ childrenReader.maxDoc(),
+ childrenReader.leaves().size());
+
+ // plain searchers without caching: AuxIndexJoinQuery is uncacheable anyway, so a cached
+ // JoinUtil query would make the comparison lopsided
+ IndexSearcher childrenSearcher = new ParallelIndexSearcher(childrenReader, executor);
+ childrenSearcher.setQueryCache(null);
+ IndexSearcher parentsSearcher = new ParallelIndexSearcher(parentsReader, executor);
+ parentsSearcher.setQueryCache(null);
+
+ Query childFilter = new TermQuery(new Term(TAG, HOT));
+ Query parentFilter = new TermQuery(new Term(COLOR, color(0)));
+
+ // the first search after a reopen builds the missing pair columns lazily (all of them in
+ // round 0, only the new segments' pairs later), so it's timed apart from the steady-state
+ // passes
+ long buildStart = System.nanoTime();
+ exactSearch(
+ parentsSearcher,
+ aiJoinChildrenToParents(joinIndex, childFilter, childrenSearcher, executor));
+ System.out.printf(
+ Locale.ROOT,
+ "AI join first search (lazy pair build): %.2fms%n",
+ (System.nanoTime() - buildStart) / 1_000_000d);
+
+ // join query creation is inside the timed task on purpose: JoinUtil runs the from-side
+ // selection eagerly in createJoinQuery, AuxIndexJoinQuery lazily in createWeight, so only
+ // create+search is comparable
+ bench(
+ "JoinUtil",
+ () -> exactSearch(parentsSearcher, joinChildrenToParents(childFilter, childrenSearcher)));
+ bench(
+ "AIJoin",
+ () ->
+ exactSearch(
+ parentsSearcher,
+ aiJoinChildrenToParents(joinIndex, childFilter, childrenSearcher, executor)));
+ bench(
+ "JoinUtil + parent filter",
+ () ->
+ exactSearch(
+ parentsSearcher,
+ filterParents(
+ joinChildrenToParents(childFilter, childrenSearcher), parentFilter)));
+ bench(
+ "AIJoin + parent filter",
+ () ->
+ exactSearch(
+ parentsSearcher,
+ filterParents(
+ aiJoinChildrenToParents(joinIndex, childFilter, childrenSearcher, executor),
+ parentFilter)));
+ }
+ }
+
+ private static Query joinChildrenToParents(Query childFilter, IndexSearcher childrenSearcher)
+ throws IOException {
+ // multipleValuesPerDocument=false picks the term-based join, no global ordinals involved
+ return JoinUtil.createJoinQuery(
+ PARENT_ID_FK, false, PARENT_ID, childFilter, childrenSearcher, ScoreMode.None);
+ }
+
+ // private static final ExecutorService executor = Executors.newFixedThreadPool(4);
+
+ private static Query aiJoinChildrenToParents(
+ AuxIndexManager joinIndex,
+ Query childFilter,
+ IndexSearcher childrenSearcher,
+ ExecutorService executor) {
+ return joinIndex.newJoinQuery(PARENT_ID_FK, childFilter, childrenSearcher, PARENT_ID, executor);
+ }
+
+ private static String color(int index) {
+ return "color" + index;
+ }
+
+ private static Query filterParents(Query joinQuery, Query parentFilter) {
+ return new BooleanQuery.Builder()
+ .add(joinQuery, BooleanClause.Occur.MUST)
+ .add(parentFilter, BooleanClause.Occur.FILTER)
+ .build();
+ }
+
+ /**
+ * Runs a top-10 search with an uncapped total hits threshold, so {@code totalHits} is always
+ * exact instead of the default two-arg {@code search(query, 10)}'s early-terminated estimate past
+ * 1000 hits: with that default, JoinUtil and AIJoin visit docs in different orders and cost
+ * estimates and stop at different points, making their reported hit counts incomparable.
+ */
+ private static TopDocs exactSearch(IndexSearcher searcher, Query query) throws IOException {
+ return searcher.search(query, new TopScoreDocCollectorManager(10, null, Integer.MAX_VALUE));
+ }
+
+ private static void bench(String name, SearchTask task) throws IOException {
+ long minNs = Long.MAX_VALUE;
+ long maxNs = 0;
+ long totalNs = 0;
+ long hits = -1;
+ for (int pass = 0; pass < PASSES; pass++) {
+ long start = System.nanoTime();
+ TopDocs topDocs = task.run();
+ long elapsed = System.nanoTime() - start;
+ minNs = Math.min(minNs, elapsed);
+ maxNs = Math.max(maxNs, elapsed);
+ totalNs += elapsed;
+ hits = topDocs.totalHits.value();
+ }
+ System.out.printf(
+ Locale.ROOT,
+ "%-26s min=%8.2fms max=%8.2fms avg=%8.2fms hits=%d%n",
+ name,
+ minNs / 1_000_000d,
+ maxNs / 1_000_000d,
+ (double) totalNs / PASSES / 1_000_000d,
+ hits);
+ }
+
+ /**
+ * Writes the two sides with plain {@link IndexWriter}s: every child points to exactly one parent,
+ * {@code parent_id} is unique on the parents side, {@code child_id} is unique on the children
+ * side, both join fields are single valued sorted string docvalues, and periodic commits under
+ * {@link NoMergePolicy} leave each side in a few segments.
+ */
+ private void buildIndices(Directory parentsDir, Directory childrenDir) throws IOException {
+ try (IndexWriter parentsWriter = newBenchWriter(parentsDir);
+ IndexWriter childrenWriter = newBenchWriter(childrenDir)) {
+ int parentsPerSegment = NUM_PARENTS / PARENT_SEGMENTS;
+ int childrenPerSegment = NUM_PARENTS * CHILDREN_PER_PARENT / CHILD_SEGMENTS;
+ int childSeq = 0;
+ for (int p = 0; p < NUM_PARENTS; p++) {
+ parentsWriter.addDocument(parentDoc(p));
+ if ((p + 1) % parentsPerSegment == 0) {
+ parentsWriter.commit();
+ }
+ for (int c = 0; c < CHILDREN_PER_PARENT; c++) {
+ childrenWriter.addDocument(childDoc(p, c));
+ if (++childSeq % childrenPerSegment == 0) {
+ childrenWriter.commit();
+ }
+ }
+ }
+ parentsWriter.commit();
+ childrenWriter.commit();
+ }
+ }
+
+ /**
+ * Simulates incremental growth after the boundary was raised to {@code numParents}: rewrites
+ * {@code numParents / }{@link #UPDATE_DENOMINATOR} parents, each with all its children, at random
+ * ids below the new boundary. Ids above the previous boundary append fresh docs; ids below it
+ * clash with existing {@code parent_id}/{@code child_id} terms and {@code updateDocument}
+ * replaces the old docs, leaving deletes behind in the earlier segments. The single closing
+ * commit adds one new segment per side under {@link NoMergePolicy}.
+ */
+ private void updateIndices(Directory parentsDir, Directory childrenDir, int numParents)
+ throws IOException {
+ try (IndexWriter parentsWriter = newBenchWriter(parentsDir);
+ IndexWriter childrenWriter = newBenchWriter(childrenDir)) {
+ for (int i = 0; i < numParents / UPDATE_DENOMINATOR; i++) {
+ int p = Math.floorMod(random, numParents);
+ random = random * 31L + 31L;
+ parentsWriter.updateDocument(new Term(PARENT_ID, parentId(p)), parentDoc(p));
+ for (int c = 0; c < CHILDREN_PER_PARENT; c++) {
+ childrenWriter.updateDocument(new Term(CHILD_ID, childId(p, c)), childDoc(p, c));
+ }
+ }
+ parentsWriter.commit();
+ childrenWriter.commit();
+ }
+ }
+
+ private IndexWriter newBenchWriter(Directory dir) throws IOException {
+ return new IndexWriter(
+ dir,
+ new IndexWriterConfig()
+ .setMergePolicy(NoMergePolicy.INSTANCE)
+ // pin the flush triggers so the intended segment layout isn't shattered by tiny
+ // default buffers
+ .setMaxBufferedDocs(IndexWriter.MAX_DOCS)
+ .setRAMBufferSizeMB(256));
+ }
+
+ private static String parentId(int p) {
+ return "parent" + p;
+ }
+
+ private static String childId(int p, int c) {
+ return "child" + p + "." + c;
+ }
+
+ private static Document parentDoc(int p) {
+ String parentId = parentId(p);
+ Document doc = new Document();
+ doc.add(new StringField(PARENT_ID, parentId, Field.Store.NO));
+ doc.add(new SortedDocValuesField(PARENT_ID, new BytesRef(parentId)));
+ // cycle colors by hot rank, not by p: a plain p % COLOR_CARDINALITY would correlate with
+ // the p % HOT_STRIDE hot selection and the filter wouldn't thin the joined parents
+ doc.add(new StringField(COLOR, color((p / HOT_STRIDE) % COLOR_CARDINALITY), Field.Store.NO));
+ return doc;
+ }
+
+ private static Document childDoc(int p, int c) {
+ Document doc = new Document();
+ doc.add(new StringField(CHILD_ID, childId(p, c), Field.Store.NO));
+ doc.add(new SortedDocValuesField(PARENT_ID_FK, new BytesRef(parentId(p))));
+ doc.add(new StringField(TAG, p % HOT_STRIDE == 0 ? HOT : COLD, Field.Store.NO));
+ return doc;
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/ParallelIndexSearcher.java b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/ParallelIndexSearcher.java
new file mode 100644
index 00000000000..eb0541554d7
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/ParallelIndexSearcher.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.util.List;
+import java.util.concurrent.Executor;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.search.IndexSearcher;
+
+final class ParallelIndexSearcher extends IndexSearcher {
+ ParallelIndexSearcher(IndexReader r, Executor executor) {
+ super(r, executor);
+ }
+
+ @Override
+ protected LeafSlice[] slices(List<LeafReaderContext> leaves) {
+ return slices(leaves, 1000, 5, false);
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoin.java b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoin.java
new file mode 100644
index 00000000000..04925e30ef2
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoin.java
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import com.carrotsearch.randomizedtesting.generators.RandomPicks;
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.UnaryOperator;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.SortedSetDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.NoMergePolicy;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.LRUQueryCache;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryCachingPolicy;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.join.JoinUtil;
+import org.apache.lucene.search.join.ScoreMode;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.IOUtils;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.jspecify.annotations.NonNull;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Joins a children index to a parents index with {@link JoinUtil}. Children reference parents via a
+ * single valued {@link SortedSetDocValuesField} {@code parent_id_FK} pointing to the parents'
+ * single valued {@code parent_id} (M:1).
+ */
+public class TestAuxIndexJoin extends SolrTestCase {
+
+ private static final String ID = "id";
+ private static final String PARENT_ID = "parent_id";
+ private static final String PARENT_ID_FK = "parent_id_FK";
+ private static final String COLOR = "color";
+
+ private static final String[] COLORS = {"red", "green", "blue"};
+ private static final int CHILDREN_PER_PARENT = 5;
+
+ private static ExecutorService executor;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @BeforeClass
+ public static void beforeClass() {
+ executor =
+ ExecutorUtil.newMDCAwareFixedThreadPool(
+ random().nextInt(1, 3),
+ new SolrNamedThreadFactory(MethodHandles.lookup().lookupClass().getSimpleName()));
+ }
+
+ @AfterClass
+ public static void afterClass() throws InterruptedException {
+ executor.shutdown();
+ assertTrue(
+ "suite-wide executor did not shut down in time",
+ executor.awaitTermination(60, TimeUnit.SECONDS));
+ }
+
+ /** Shared per-test auxiliary join index: pair columns are built lazily by the first search. */
+ private AuxIndexManager joinIndex;
+
+ private Directory joinDir;
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ joinDir = newDirectory();
+ // none of these affect join correctness, only internal segment layout, refresh latency, and
+ // dead-pair reap timing -- randomized to exercise all combinations
+ AuxIndexJoinConfig config =
+ new AuxIndexJoinConfig()
+ .setSingleFieldPerSegment(random().nextBoolean())
+ .setBlockingRefresh(random().nextBoolean())
+ .setUseFromSideThreads(random().nextBoolean())
+ .setSweepSamplingInterval(TestUtil.nextInt(random(), -1, 2), TimeUnit.MINUTES)
+ // 0 commits every batch, 30s leaves them flushed but uncommitted for the whole
+ // run -- correctness must not be able to tell the two apart
+ .setCommitIntervalMs(random().nextBoolean() ? 0 : TimeUnit.SECONDS.toMillis(30));
+ joinIndex = new AuxIndexManager(joinDir, config);
+ }
+
+ @Override
+ public void tearDown() throws Exception {
+ IOUtils.close(joinIndex, joinDir);
+ super.tearDown();
+ }
+
+ /** Two indices: parents and their children, segmented by intermediate commits. */
+ private static class ParentChildIndices implements Closeable {
+ final Directory parentsDir = newDirectory();
+ final Directory childrenDir = newDirectory();
+ final RandomIndexWriter parentsWriter;
+ final RandomIndexWriter childrenWriter;
+ // childId -> parentId, insertion order
+ final Map<String, String> parentIdByChildId = new TreeMap<>();
+ final Map<String, String> colorByParentId = new TreeMap<>();
+
+ ParentChildIndices() throws IOException {
+ // NoMergePolicy keeps the segments created by the intermediate commits below
+ parentsWriter =
+ new RandomIndexWriter(
+ random(),
+ parentsDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE));
+ childrenWriter =
+ new RandomIndexWriter(
+ random(),
+ childrenDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE));
+ int numParents = atLeast(10);
+ int childSeq = 0;
+ for (int p = 0; p < numParents; p++) {
+ String parentId = "parent" + p;
+ String color = RandomPicks.randomFrom(random(), COLORS);
+ colorByParentId.put(parentId, color);
+ parentsWriter.addDocument(parentDoc(parentId, color));
+ for (int c = 0; c < CHILDREN_PER_PARENT; c++) {
+ String childId = "child" + childSeq++;
+ parentIdByChildId.put(childId, parentId);
+ childrenWriter.addDocument(childDoc(childId, parentId));
+ }
+ if (p % 3 == 2) {
+ parentsWriter.commit();
+ childrenWriter.commit();
+ }
+ }
+ }
+
+ List<String> childrenOf(String parentId) {
+ List<String> children = new ArrayList<>();
+ parentIdByChildId.forEach(
+ (childId, pid) -> {
+ if (pid.equals(parentId)) {
+ children.add(childId);
+ }
+ });
+ return children;
+ }
+
+ @Override
+ public void close() throws IOException {
+ IOUtils.close(parentsWriter, childrenWriter, parentsDir, childrenDir);
+ }
+ }
+
+ private static Document parentDoc(String parentId, String color) {
+ Document doc = new Document();
+ doc.add(new StringField(PARENT_ID, parentId, Field.Store.YES));
+ doc.add(new SortedSetDocValuesField(PARENT_ID, new BytesRef(parentId)));
+ doc.add(new StringField(COLOR, color, Field.Store.YES));
+ return doc;
+ }
+
+ private static Document childDoc(String childId, String parentId) {
+ Document doc = new Document();
+ doc.add(new StringField(ID, childId, Field.Store.YES));
+ doc.add(new StringField(PARENT_ID_FK, parentId, Field.Store.YES));
+ doc.add(new SortedSetDocValuesField(PARENT_ID_FK, new BytesRef(parentId)));
+ return doc;
+ }
+
+ private static Query joinChildrenToParents(Query fromQuery, IndexSearcher childrenSearcher)
+ throws IOException {
+ // any score mode matches the same parents
+ ScoreMode scoreMode = RandomPicks.randomFrom(random(), ScoreMode.values());
+ return JoinUtil.createJoinQuery(
+ PARENT_ID_FK, true, PARENT_ID, fromQuery, childrenSearcher, scoreMode);
+ }
+
+ private static Set<String> searchParentIds(IndexSearcher parentsSearcher, Query query)
+ throws IOException {
+ TopDocs topDocs = parentsSearcher.search(query, parentsSearcher.getIndexReader().maxDoc());
+ Set<String> parentIds = new TreeSet<>();
+ for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
+ parentIds.add(parentsSearcher.storedFields().document(scoreDoc.doc).get(PARENT_ID));
+ }
+ return parentIds;
+ }
+
+ /**
+ * Searches the same join through {@link JoinUtil} and through the auxiliary join index and
+ * asserts both agree; {@code joinDecorator} wraps each join query the same way before searching.
+ */
+ private Set<String> searchParentIdsBothJoins(
+ IndexSearcher parentsSearcher,
+ Query fromQuery,
+ IndexSearcher childrenSearcher,
+ UnaryOperator<Query> joinDecorator)
+ throws IOException {
+ Query joinUtilQuery = joinChildrenToParents(fromQuery, childrenSearcher);
+ Set<String> joinUtilParents =
+ searchParentIds(parentsSearcher, joinDecorator.apply(joinUtilQuery));
+ Query aiJoinQuery = createAiJoinQuery(joinIndex, fromQuery, childrenSearcher);
+ assertEquals(
+ "AuxIndexJoinQuery disagrees with JoinUtil",
+ joinUtilParents,
+ searchParentIds(parentsSearcher, joinDecorator.apply(aiJoinQuery)));
+ return joinUtilParents;
+ }
+
+ private @NonNull Query createAiJoinQuery(
+ AuxIndexManager joinIndexParam, Query fromQuery, IndexSearcher childrenSearcher) {
+ return joinIndexParam.newJoinQuery(
+ PARENT_ID_FK,
+ fromQuery,
+ random().nextBoolean() ? cachedSearcher(childrenSearcher) : childrenSearcher,
+ PARENT_ID,
+ rarely() ? null : executor);
+ }
+
+ private Set<String> searchParentIdsBothJoins(
+ IndexSearcher parentsSearcher, Query fromQuery, IndexSearcher childrenSearcher)
+ throws IOException {
+ return searchParentIdsBothJoins(
+ parentsSearcher, fromQuery, childrenSearcher, UnaryOperator.identity());
+ }
+
+ private static Query anyOfChildren(Set<String> childIds) {
+ BooleanQuery.Builder builder = new BooleanQuery.Builder();
+ for (String childId : childIds) {
+ builder.add(new TermQuery(new Term(ID, childId)), BooleanClause.Occur.SHOULD);
+ }
+ return builder.build();
+ }
+
+ private static Set<String> randomChildrenSubset(ParentChildIndices indices) {
+ Set<String> selected = new TreeSet<>();
+ for (String childId : indices.parentIdByChildId.keySet()) {
+ if (random().nextBoolean()) {
+ selected.add(childId);
+ }
+ }
+ if (selected.isEmpty()) {
+ selected.add(RandomPicks.randomFrom(random(), indices.parentIdByChildId.keySet()));
+ }
+ return selected;
+ }
+
+ private static Set<String> expectedParents(ParentChildIndices indices, Set<String> childIds) {
+ Set<String> expected = new TreeSet<>();
+ for (String childId : childIds) {
+ expected.add(indices.parentIdByChildId.get(childId));
+ }
+ return expected;
+ }
+
+ public void testAIJoinRandomChildrenSubset() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ assertTrue("children index should be segmented", childrenReader.leaves().size() > 1);
+ assertTrue("parents index should be segmented", parentsReader.leaves().size() > 1);
+
+ Set<String> selectedChildren = randomChildrenSubset(indices);
+ // the first search builds the missing pair columns into the join index on demand
+ assertEquals(
+ expectedParents(indices, selectedChildren),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ anyOfChildren(selectedChildren),
+ newSearcher(childrenReader)));
+ }
+
+ // grow both sides and reopen: the next search finds the old pair columns persisted and
+ // lazily builds only the pairs involving the new segments
+ String newParentId = "parentNew";
+ String newParentColor = RandomPicks.randomFrom(random(), COLORS);
+ indices.colorByParentId.put(newParentId, newParentColor);
+ indices.parentsWriter.addDocument(parentDoc(newParentId, newParentColor));
+ String newChildId = "childNew";
+ String oldChildId = RandomPicks.randomFrom(random(), indices.parentIdByChildId.keySet());
+ indices.parentIdByChildId.put(newChildId, newParentId);
+ indices.childrenWriter.addDocument(childDoc(newChildId, newParentId));
+ indices.parentsWriter.commit();
+ indices.childrenWriter.commit();
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ assertEquals(
+ Set.of(newParentId),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ new TermQuery(new Term(ID, newChildId)),
+ newSearcher(childrenReader)));
+
+ // one query spanning old and new children joins to their old and new parents
+ assertEquals(
+ Set.of(indices.parentIdByChildId.get(oldChildId), newParentId),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ anyOfChildren(new TreeSet<>(List.of(oldChildId, newChildId))),
+ newSearcher(childrenReader)));
+ }
+ }
+ }
+
+ /**
+ * A from-side segment whose join field holds a single distinct value (one parent referenced by
+ * every child) must still answer lookups for to-side terms absent from it. The per-segment term
+ * dictionary then has {@code getValueCount() == 1}, which used to seed the lookup hash at size 1
+ * -- a table {@code add()} fills to 100%, so probing for the orphan parent's term never found an
+ * empty slot and spun forever.
+ */
+ public void testFromSegmentWithSingleDistinctValue() throws Exception {
+ try (Directory childrenDir = newDirectory();
+ Directory parentsDir = newDirectory()) {
+ // a plain IndexWriter (not RandomIndexWriter) keeps all children in one in-RAM segment until
+ // close, so the from-segment's term dictionary has exactly one distinct value
+ try (IndexWriter childrenWriter =
+ new IndexWriter(
+ childrenDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE))) {
+ for (int c = 0; c < 5; c++) {
+ childrenWriter.addDocument(childDoc("child" + c, "onlyParent"));
+ }
+ }
+ try (IndexWriter parentsWriter =
+ new IndexWriter(
+ parentsDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE))) {
+ // one parent referenced by every child, plus an orphan parent whose term is absent from the
+ // children's single-value term dictionary -- the lookup that used to hang
+ parentsWriter.addDocument(parentDoc("onlyParent", "red"));
+ parentsWriter.addDocument(parentDoc("orphanParent", "blue"));
+ }
+ try (IndexReader childrenReader = DirectoryReader.open(childrenDir);
+ IndexReader parentsReader = DirectoryReader.open(parentsDir)) {
+ assertEquals(
+ Set.of("onlyParent"),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ anyOfChildren(new TreeSet<>(List.of("child0", "child1"))),
+ newSearcher(childrenReader)));
+ }
+ }
+ }
+
+ public void testUpdateChildParentIdFK() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ // repeated re-pointing exercises the sidecar join index's update path many times over,
+ // not just a single one-off update
+ for (int iter = 0; iter < atLeast(5); iter++) {
+ String childId = RandomPicks.randomFrom(random(), indices.parentIdByChildId.keySet());
+ String oldParentId = indices.parentIdByChildId.get(childId);
+ String newParentId;
+ do {
+ newParentId = RandomPicks.randomFrom(random(), indices.colorByParentId.keySet());
+ } while (newParentId.equals(oldParentId));
+
+ // SortedSetDocValues can't be updated in place, so replace the whole child document
+ indices.childrenWriter.updateDocument(
+ new Term(ID, childId), childDoc(childId, newParentId));
+ indices.parentIdByChildId.put(childId, newParentId);
+
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ IndexSearcher childrenSearcher = newSearcher(childrenReader);
+ IndexSearcher parentsSearcher = newSearcher(parentsReader);
+
+ assertEquals(
+ Set.of(newParentId),
+ searchParentIdsBothJoins(
+ parentsSearcher, new TermQuery(new Term(ID, childId)), childrenSearcher));
+
+ // the old parent is reachable only while it still has other children pointing at it
+ Set<String> expectedOldParent =
+ indices.childrenOf(oldParentId).isEmpty() ? Set.of() : Set.of(oldParentId);
+ assertEquals(
+ expectedOldParent,
+ searchParentIdsBothJoins(
+ parentsSearcher,
+ new TermQuery(new Term(PARENT_ID_FK, oldParentId)),
+ childrenSearcher));
+ }
+ }
+ }
+ }
+
+ public void testUpdateParentId() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ // repeated renaming exercises the sidecar join index's update path many times over, not
+ // just a single one-off rename
+ for (int iter = 0; iter < atLeast(5); iter++) {
+ String parentId = RandomPicks.randomFrom(random(), indices.colorByParentId.keySet());
+ String renamedParentId = parentId + "-renamed" + iter;
+ String color = indices.colorByParentId.remove(parentId);
+ indices.parentsWriter.updateDocument(
+ new Term(PARENT_ID, parentId), parentDoc(renamedParentId, color));
+ indices.colorByParentId.put(renamedParentId, color);
+
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ // children still point at the old id, so they join to nothing
+ assertEquals(
+ Set.of(),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ new TermQuery(new Term(PARENT_ID_FK, parentId)),
+ newSearcher(childrenReader)));
+ }
+
+ // re-point one child at the renamed parent and the join works again, if any of this
+ // parent's children haven't already been re-pointed elsewhere by an earlier iteration
+ List<String> children = indices.childrenOf(parentId);
+ if (children.isEmpty()) {
+ continue;
+ }
+ String childId = RandomPicks.randomFrom(random(), children);
+ indices.childrenWriter.updateDocument(
+ new Term(ID, childId), childDoc(childId, renamedParentId));
+ indices.parentIdByChildId.put(childId, renamedParentId);
+
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ assertEquals(
+ Set.of(renamedParentId),
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ new TermQuery(new Term(ID, childId)),
+ newSearcher(childrenReader)));
+ }
+ }
+ }
+ }
+
+ public void testAIJoinWithParentTermFilter() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader();
+ Directory joinDir = newDirectory();
+ AuxIndexManager joinIndex = new AuxIndexManager(joinDir)) {
+ Set<String> selectedChildren = randomChildrenSubset(indices);
+ String color = RandomPicks.randomFrom(random(), COLORS);
+
+ Query aiJoinQuery =
+ createAiJoinQuery(
+ joinIndex, anyOfChildren(selectedChildren), newSearcher(childrenReader));
+
+ Query filteredJoin =
+ new BooleanQuery.Builder()
+ .add(aiJoinQuery, BooleanClause.Occur.MUST)
+ .add(new TermQuery(new Term(COLOR, color)), BooleanClause.Occur.FILTER)
+ .build();
+
+ Set<String> expected = new TreeSet<>();
+ for (String parentId : expectedParents(indices, selectedChildren)) {
+ if (color.equals(indices.colorByParentId.get(parentId))) {
+ expected.add(parentId);
+ }
+ }
+ assertEquals(expected, searchParentIds(newSearcher(parentsReader), filteredJoin));
+ }
+ }
+ }
+
+ /**
+ * Not a correctness test but a tracing scenario for the repeated-ForeignKeyColumn-load issue;
+ * asserts nothing, read the {@code AIJOIN} log lines instead.
+ *
+ * <p>The FILTER clause is added <b>before</b> the join clause and filters on {@code
+ * parent_id=parent0}, a term present in only the first parents segment. Per to-leaf, {@code
+ * BooleanWeight.scorerSupplier} asks the clauses in order and bails out with {@code null} on the
+ * first required clause that has no matches -- so for every to-segment except the first one the
+ * join's scorerSupplier is never invoked, no {@code JoinIndexScorerSupplier} is created, and the
+ * pairs (every from-segment x that to-segment) are never written to the sidecar.
+ *
+ * <p>Expected log shape: pass 1 loads all FK columns and builds only the first to-segment's
+ * pairs; passes 2..3 show {@code evt=weight} with the same {@code pairsMissing} again, {@code
+ * evt=fkload} repeating for every from-segment, and no {@code evt=build} -- the profiler-visible
+ * waste. The unfiltered pass then visits every to-segment and writes the remaining pairs, after
+ * which the final filtered pass runs with {@code pairsMissing=0} and no {@code evt=fkload}.
+ */
+ @Ignore
+ public void testTraceFkReloadForSkippedToSegments() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ IndexSearcher parentsSearcher = newSearcher(parentsReader);
+ Query aiJoinQuery =
+ createAiJoinQuery(
+ joinIndex,
+ anyOfChildren(randomChildrenSubset(indices)),
+ newSearcher(childrenReader));
+ Query filteredJoin =
+ new BooleanQuery.Builder()
+ // FILTER first: a to-segment without the term short-circuits BooleanWeight
+ // before the join clause's scorerSupplier is consulted
+ .add(new TermQuery(new Term(PARENT_ID, "parent0")), BooleanClause.Occur.FILTER)
+ .add(aiJoinQuery, BooleanClause.Occur.MUST)
+ .build();
+ for (int pass = 1; pass <= 3; pass++) {
+ log.info(
+ "AIJOIN-TRACE pass={} filtered join; beyond pass 1 expect repeated pairsMissing"
+ + " and evt=fkload with no evt=build",
+ pass);
+ searchParentIds(parentsSearcher, filteredJoin);
+ }
+ log.info(
+ "AIJOIN-TRACE unfiltered join: every to-segment is visited, remaining pairs written");
+ searchParentIds(parentsSearcher, aiJoinQuery);
+ log.info("AIJOIN-TRACE final filtered pass: expect pairsMissing=0 and no evt=fkload");
+ searchParentIds(parentsSearcher, filteredJoin);
+ }
+ }
+ }
+
+ public void testJoinWithParentTermFilter() throws Exception {
+ try (ParentChildIndices indices = new ParentChildIndices()) {
+ try (IndexReader childrenReader = indices.childrenWriter.getReader();
+ IndexReader parentsReader = indices.parentsWriter.getReader()) {
+ Set<String> selectedChildren = randomChildrenSubset(indices);
+ String color = RandomPicks.randomFrom(random(), COLORS);
+
+ Set<String> expected = new TreeSet<>();
+ for (String parentId : expectedParents(indices, selectedChildren)) {
+ if (color.equals(indices.colorByParentId.get(parentId))) {
+ expected.add(parentId);
+ }
+ }
+ assertEquals(
+ expected,
+ searchParentIdsBothJoins(
+ newSearcher(parentsReader),
+ anyOfChildren(selectedChildren),
+ newSearcher(childrenReader),
+ join ->
+ new BooleanQuery.Builder()
+ .add(join, BooleanClause.Occur.MUST)
+ .add(new TermQuery(new Term(COLOR, color)), BooleanClause.Occur.FILTER)
+ .build()));
+ }
+ }
+ }
+
+ // presumably keep it in AuxIndexManager
+ private static IndexSearcher cachedSearcher(IndexSearcher fromSearcher) {
+ IndexSearcher cachedFromSearcher =
+ new IndexSearcher(
+ fromSearcher.getIndexReader() // , executor // pointless
+ );
+ cachedFromSearcher.setQueryCache(
+ new LRUQueryCache(
+ fromSearcher.getLeafContexts().size() + 1,
+ fromSearcher.getIndexReader().maxDoc() / 8 * 2));
+ cachedFromSearcher.setQueryCachingPolicy(
+ new QueryCachingPolicy() {
+ @Override
+ public boolean shouldCache(Query query) {
+ return true;
+ }
+
+ @Override
+ public void onUse(Query query) {}
+ });
+ return cachedFromSearcher;
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoinMergePolicy.java b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoinMergePolicy.java
new file mode 100644
index 00000000000..64da79ca630
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestAuxIndexJoinMergePolicy.java
@@ -0,0 +1,990 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.SortedSetDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.IndexNotFoundException;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NoMergePolicy;
+import org.apache.lucene.index.SegmentCommitInfo;
+import org.apache.lucene.index.SegmentInfos;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.index.TieredMergePolicy;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.ScoreMode;
+import org.apache.lucene.search.ScorerSupplier;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.Weight;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.IOUtils;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.Edges;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+
+/**
+ * Real (non-mocked) integration smoke test for {@link AuxIndexJoinMergePolicy}: builds a segmented
+ * children/parents pair, forces {@link AuxIndexJoinQuery} to populate the sidecar join index, then
+ * mass deletes and force-merges both sides so the sidecar's previously-built pair columns go stale,
+ * and checks the policy actually notices and reaps them -- not just that it runs without throwing.
+ */
+public class TestAuxIndexJoinMergePolicy extends SolrTestCase {
+
+ private static final String ID = "id";
+ private static final String PARENT_ID = "parent_id";
+ private static final String PARENT_ID_FK = "parent_id_FK";
+
+ private Directory parentsDir;
+ private Directory childrenDir;
+ private Directory joinDir;
+ private RandomIndexWriter parentsWriter;
+ private RandomIndexWriter childrenWriter;
+ private AuxIndexManager joinIndex;
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ parentsDir = newDirectory();
+ childrenDir = newDirectory();
+ // NoMergePolicy keeps every intermediate commit as its own segment, so the join index has
+ // several distinct (from-segment, to-segment) pairs to build, not just one
+ parentsWriter =
+ new RandomIndexWriter(
+ random(),
+ parentsDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE));
+ childrenWriter =
+ new RandomIndexWriter(
+ random(),
+ childrenDir,
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE));
+ joinDir = newDirectory();
+ // every assertion here reads the sidecar off the directory rather than through the manager's
+ // NRT searcher, so the batches have to be committed, not merely flushed
+ joinIndex = new AuxIndexManager(joinDir, commitEveryBatch());
+ // this test drives many onCreateWeight calls back to back, well inside the default one-minute
+ // sampling interval, and asserts on the reaper noticing every one of them
+ joinIndex.mergePolicy.setSweepInterval(0, TimeUnit.NANOSECONDS);
+ // the sidecars here are kilobytes, so the default megabyte of reclaimable bytes would decline
+ // every purge; the two tests below that are about the threshold itself set their own
+ joinIndex.mergePolicy.setMinReclaimableBytesToPurge(0);
+ }
+
+ @Override
+ public void tearDown() throws Exception {
+ IOUtils.close(joinIndex, joinDir, parentsWriter, childrenWriter, parentsDir, childrenDir);
+ super.tearDown();
+ }
+
+ private static Document parentDoc(String parentId) {
+ Document doc = new Document();
+ doc.add(new StringField(PARENT_ID, parentId, Field.Store.YES));
+ doc.add(new SortedSetDocValuesField(PARENT_ID, new BytesRef(parentId)));
+ return doc;
+ }
+
+ private static Document childDoc(String childId, String parentId) {
+ Document doc = new Document();
+ doc.add(new StringField(ID, childId, Field.Store.YES));
+ doc.add(new StringField(PARENT_ID_FK, parentId, Field.Store.YES));
+ doc.add(new SortedSetDocValuesField(PARENT_ID_FK, new BytesRef(parentId)));
+ return doc;
+ }
+
+ /**
+ * Adds {@code numParents} parents (3 children each), committing every 3 parents so both sides end
+ * up segmented rather than a single flush. Returns the added parent ids.
+ */
+ private List<String> addParentsAndChildren(String parentIdPrefix, int numParents)
+ throws IOException {
+ List<String> parentIds = new ArrayList<>();
+ for (int p = 0; p < numParents; p++) {
+ String parentId = parentIdPrefix + p;
+ parentIds.add(parentId);
+ parentsWriter.addDocument(parentDoc(parentId));
+ for (int c = 0; c < 3; c++) {
+ childrenWriter.addDocument(childDoc(parentId + "_child" + c, parentId));
+ }
+ if (p % 3 == 2) {
+ parentsWriter.commit();
+ childrenWriter.commit();
+ }
+ }
+ parentsWriter.commit();
+ childrenWriter.commit();
+ return parentIds;
+ }
+
+ /**
+ * Deletes every parent in {@code parentIds} (and its children), then force-merges both sides down
+ * to a single segment each -- changing both sides' segment identities so every sidecar pair
+ * column referencing them goes stale.
+ */
+ private void deleteParentsAndForceMerge(List<String> parentIds) throws IOException {
+ for (String parentId : parentIds) {
+ parentsWriter.deleteDocuments(new Term(PARENT_ID, parentId));
+ for (int c = 0; c < 3; c++) {
+ childrenWriter.deleteDocuments(new Term(ID, parentId + "_child" + c));
+ }
+ }
+ // NoMergePolicy (set at construction, to guarantee segmentation above) blocks forceMerge too,
+ // so swap in a real policy just for these force merges
+ parentsWriter.w.getConfig().setMergePolicy(new TieredMergePolicy());
+ childrenWriter.w.getConfig().setMergePolicy(new TieredMergePolicy());
+ parentsWriter.forceMerge(1);
+ childrenWriter.forceMerge(1);
+ parentsWriter.commit();
+ childrenWriter.commit();
+ }
+
+ /**
+ * Runs the join query for every child against every parent and returns the matched parent ids,
+ * forcing {@link AuxIndexJoinQuery} to fully execute (not just build its {@link
+ * org.apache.lucene.search.Weight}) so missing pair columns actually get built into the sidecar.
+ */
+ private Set<String> searchAllParents(
+ IndexSearcher parentsSearcher, IndexSearcher childrenSearcher) throws IOException {
+ Query aiJoinQuery =
+ joinIndex.newJoinQuery(PARENT_ID_FK, new MatchAllDocsQuery(), childrenSearcher, PARENT_ID);
+ TopDocs topDocs =
+ parentsSearcher.search(aiJoinQuery, parentsSearcher.getIndexReader().maxDoc());
+ Set<String> parentIds = new TreeSet<>();
+ for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
+ parentIds.add(parentsSearcher.storedFields().document(scoreDoc.doc).get(PARENT_ID));
+ }
+ return parentIds;
+ }
+
+ public void testDeadPairsAreReapedAfterForceMerge() throws Exception {
+ List<String> firstBatch = addParentsAndChildren("gen1-", atLeast(15));
+
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ assertTrue(
+ "children index should be segmented before merging", childrenReader.leaves().size() > 1);
+ assertTrue(
+ "parents index should be segmented before merging", parentsReader.leaves().size() > 1);
+
+ // populates the sidecar with every (child-segment, parent-segment) pair currently needed
+ assertEquals(
+ new TreeSet<>(firstBatch),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+
+ assertEquals(
+ "nothing should look dead yet, nothing has disappeared",
+ 0,
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+ assertEquals(0, joinIndex.mergePolicy.droppedSegmentCount());
+
+ // mass removal: drop every other parent (and its children), then force-merge both sides --
+ // every pair column built above referenced a from/to segment that no longer exists
+ List<String> deadFromFirstBatch = new ArrayList<>();
+ List<String> survivingParents = new ArrayList<>();
+ for (int i = 0; i < firstBatch.size(); i++) {
+ (i % 2 == 0 ? deadFromFirstBatch : survivingParents).add(firstBatch.get(i));
+ }
+ deleteParentsAndForceMerge(deadFromFirstBatch);
+
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ assertEquals(1, childrenReader.leaves().size());
+ assertEquals(1, parentsReader.leaves().size());
+
+ // re-running the same join rebuilds pairs against the new segments and, in the process,
+ // notices the old pairs vanished
+ assertEquals(
+ new TreeSet<>(survivingParents),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+
+ joinIndex.waitForMerges();
+
+ // reapedPairCount, not pendingPairRemovalsCount: the queue is what is still *owed*, and now
+ // that reaping actually drains it, it settles back to zero. What has to be true is that the
+ // dead pairs were recognized and then reclaimed, which is what this counts.
+ assertTrue(
+ "the old pair columns should have been recognized as dead and reaped",
+ joinIndex.mergePolicy.reapedPairCount() > 0);
+ assertTrue(
+ "AuxIndexJoinMergePolicy should have reaped at least the one dead sidecar segment",
+ joinIndex.mergePolicy.droppedSegmentCount() >= 1);
+
+ // do it again: grow, mass-delete, force-merge -- the reaper should keep working, not just
+ // fire once. Force-merging an index that already sits at a single, fully-live segment is a
+ // no-op (nothing to reclaim), so the surviving segment above keeps its identity and its
+ // sidecar pair never goes stale on its own; adding one more small batch afterwards guarantees
+ // a genuinely new pair needs building, which is what actually triggers the sidecar's next
+ // commit (and so the next reap opportunity) -- the reaper is piggybacked on writes, not on a
+ // background timer.
+ int droppedSoFar = joinIndex.mergePolicy.droppedSegmentCount();
+ int reapedSoFar = joinIndex.mergePolicy.reapedPairCount();
+ List<String> secondBatch = addParentsAndChildren("gen2-", atLeast(15));
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ Set<String> expected = new TreeSet<>(survivingParents);
+ expected.addAll(secondBatch);
+ assertEquals(
+ expected, searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+
+ deleteParentsAndForceMerge(secondBatch);
+ List<String> thirdBatch = addParentsAndChildren("gen3-", 1);
+
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ Set<String> expected = new TreeSet<>(survivingParents);
+ expected.addAll(thirdBatch);
+ assertEquals(
+ expected, searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+
+ joinIndex.waitForMerges();
+
+ // more dead *pairs* reclaimed, not necessarily more dead *segments*: which of the two routes
+ // reaping takes depends on whether a segment's pairs all die together, and with column purging
+ // available a segment often sheds its dead columns before it can become entirely dead
+ assertTrue(
+ "a second round of mass deletes + force merge should reclaim more dead pairs; dropped "
+ + droppedSoFar
+ + "->"
+ + joinIndex.mergePolicy.droppedSegmentCount()
+ + " purged "
+ + joinIndex.mergePolicy.purgedSegmentCount()
+ + " reaped pairs "
+ + reapedSoFar
+ + "->"
+ + joinIndex.mergePolicy.reapedPairCount(),
+ joinIndex.mergePolicy.reapedPairCount() > reapedSoFar);
+ }
+
+ /**
+ * Writes {@code batches} single-pair batches straight into the sidecar -- one segment each, of
+ * deliberately differing lengths -- and returns the from-doc -> to-doc column each pair was
+ * written with, keyed by pair field name.
+ */
+ private Map<String, int[]> writeBatchesOfVaryingLength(int firstBatch, int batches)
+ throws IOException {
+ Map<String, int[]> columnsByPair = new LinkedHashMap<>();
+ for (int batch = firstBatch; batch < firstBatch + batches; batch++) {
+ // the shape of a real pair field name doesn't matter here, only that it is unique per pair
+ String pairFieldName = "fromSeg" + batch + ":dv0_toSeg:dv0";
+ int fromSegmentMaxDoc = 2 + batch; // pairs come from from-segments of different sizes
+ int[] toDocByFromDoc = new int[fromSegmentMaxDoc];
+ Arrays.fill(toDocByFromDoc, -1);
+ // every other from-doc matches, starting at doc 0 for even batches and doc 1 for odd ones,
+ // so the union has to keep both "doc 0 has a value" and "doc 0 has none" columns straight
+ for (int fromDoc = batch % 2; fromDoc < fromSegmentMaxDoc; fromDoc += 2) {
+ toDocByFromDoc[fromDoc] = 1000 * batch + fromDoc;
+ }
+ columnsByPair.put(pairFieldName, toDocByFromDoc);
+ joinIndex.writeBatch(Map.of(pairFieldName, model(toDocByFromDoc)));
+ }
+ return columnsByPair;
+ }
+
+ /** The {@link JoinColumnModel} a from-doc -> to-doc array implies, edges included. */
+ private static JoinColumnModel model(int[] toDocByFromDoc) {
+ int minFromDoc = -1;
+ int maxFromDoc = -1;
+ int minToDoc = -1;
+ int maxToDoc = -1;
+ int toCount = 0;
+ for (int fromDoc = 0; fromDoc < toDocByFromDoc.length; fromDoc++) {
+ int toDoc = toDocByFromDoc[fromDoc];
+ if (toDoc < 0) {
+ continue;
+ }
+ if (toCount == 0) {
+ minFromDoc = fromDoc;
+ minToDoc = toDoc;
+ }
+ maxFromDoc = fromDoc;
+ minToDoc = Math.min(minToDoc, toDoc);
+ maxToDoc = Math.max(maxToDoc, toDoc);
+ toCount++;
+ }
+ return JoinColumnModel.dense(
+ toDocByFromDoc,
+ new Edges(new int[] {minFromDoc, maxFromDoc}, new int[] {minToDoc, maxToDoc}, toCount));
+ }
+
+ /**
+ * Asserts the pair's column reads back from the sidecar exactly as written: same to-doc at the
+ * same from-doc, nothing where nothing was written (padding included), and its edges still at doc
+ * 0 -- i.e. that compaction moved no doc id.
+ */
+ private static void assertColumnIntact(IndexReader sidecar, String pairFieldName, int[] expected)
+ throws IOException {
+ String toCountField = JoinIndexUtils.TO_COUNT_PREFIX + pairFieldName;
+ LeafReaderContext carrier = null;
+ for (LeafReaderContext leaf : sidecar.leaves()) {
+ if (leaf.reader().getFieldInfos().fieldInfo(toCountField) != null) {
+ assertNull("pair " + pairFieldName + " ended up in two sidecar segments", carrier);
+ carrier = leaf;
+ }
+ }
+ assertNotNull("pair " + pairFieldName + " was lost", carrier);
+
+ JoinColumnModel written = model(expected);
+ assertArrayEquals(
+ "from-doc edges of " + pairFieldName,
+ written.edges().fromDocEdges(),
+ JoinIndexUtils.loadEdges(carrier, JoinIndexUtils.FROM_EDGES_PREFIX + pairFieldName));
+ assertArrayEquals(
+ "to-doc edges of " + pairFieldName,
+ written.edges().toDocEdges(),
+ JoinIndexUtils.loadEdges(carrier, JoinIndexUtils.TO_EDGES_PREFIX + pairFieldName));
+ assertArrayEquals(
+ "to-doc count of " + pairFieldName,
+ new int[] {written.edges().toCount()},
+ JoinIndexUtils.loadEdges(carrier, toCountField));
+
+ SortedNumericDocValues column =
+ carrier
+ .reader()
+ .getSortedNumericDocValues(JoinIndexUtils.TO_DOC_VAL_BY_FROM_DOCNUM + pairFieldName);
+ assertNotNull("join column of " + pairFieldName, column);
+ for (int fromDoc = 0; fromDoc < expected.length; fromDoc++) {
+ String at = pairFieldName + " at from-doc " + fromDoc;
+ assertEquals(at, expected[fromDoc] >= 0, column.advanceExact(fromDoc));
+ if (expected[fromDoc] >= 0) {
+ assertEquals(at, 1, column.docValueCount());
+ assertEquals(at, expected[fromDoc], column.nextValue());
+ }
+ }
+ // the tail this pair's from-segment never had: padded by the merge, and still empty
+ for (int paddedDoc = expected.length; paddedDoc < carrier.reader().maxDoc(); paddedDoc++) {
+ assertFalse(
+ pairFieldName + " has a value at padded doc " + paddedDoc,
+ column.advanceExact(paddedDoc));
+ }
+ }
+
+ /**
+ * The sidecar is written one batch per segment and can never be merged the ordinary way, so
+ * compaction has to fold segments together by doc id. Checks it happens, and that every column
+ * still answers at exactly the from-doc it was written at.
+ */
+ public void testDocAlignedMergeUnionsColumnsWithoutMovingDocIds() throws Exception {
+ // fold every 3 segments, so a handful of batches is enough to see compaction converge
+ joinIndex.mergePolicy.setCompaction(3, AuxIndexJoinMergePolicy.DEFAULT_MAX_PAIRS_PER_SEGMENT);
+
+ Map<String, int[]> columnsByPair = writeBatchesOfVaryingLength(0, 9);
+ joinIndex.waitForMerges();
+ // one more batch, so the commit it makes publishes the merged segments to the reader below
+ columnsByPair.putAll(writeBatchesOfVaryingLength(9, 1));
+
+ assertTrue(
+ "expected the sidecar's segments to be compacted, none were",
+ joinIndex.mergePolicy.alignedMergeCount() > 0);
+
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertTrue(
+ "compaction should have left fewer segments than the "
+ + columnsByPair.size()
+ + " written batches, got "
+ + sidecar.leaves().size(),
+ sidecar.leaves().size() < columnsByPair.size());
+ for (Map.Entry<String, int[]> pair : columnsByPair.entrySet()) {
+ assertColumnIntact(sidecar, pair.getKey(), pair.getValue());
+ }
+ }
+ }
+
+ /** The same, end to end: a join keeps answering identically once its sidecar is compacted. */
+ public void testJoinAnswersTheSameAfterCompaction() throws Exception {
+ joinIndex.mergePolicy.setCompaction(2, AuxIndexJoinMergePolicy.DEFAULT_MAX_PAIRS_PER_SEGMENT);
+
+ List<String> parentIds = addParentsAndChildren("gen1-", atLeast(15));
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ // builds the sidecar: one segment per written batch, several pairs' worth
+ assertEquals(
+ new TreeSet<>(parentIds),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+
+ joinIndex.waitForMerges();
+ assertTrue(
+ "expected the sidecar's segments to be compacted, none were",
+ joinIndex.mergePolicy.alignedMergeCount() > 0);
+
+ // same query, same answer -- now served from doc-aligned, compacted sidecar segments
+ assertEquals(
+ new TreeSet<>(parentIds),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+ }
+
+ /**
+ * Writes one batch carrying {@code pairCount} pairs -- so one sidecar segment {@code pairCount}
+ * columns wide, which is what the purge threshold measures against. Returns its columns by pair
+ * name.
+ */
+ private Map<String, int[]> writeWideBatch(int batch, int pairCount) throws IOException {
+ Map<String, int[]> columnsByPair = new LinkedHashMap<>();
+ Map<String, JoinColumnModel> written = new LinkedHashMap<>();
+ for (int slot = 0; slot < pairCount; slot++) {
+ String pairFieldName = "fromSeg" + batch + "-" + slot + ":dv0_toSeg:dv0";
+ int[] toDocByFromDoc = new int[2 + batch];
+ Arrays.fill(toDocByFromDoc, -1);
+ for (int fromDoc = slot % 2; fromDoc < toDocByFromDoc.length; fromDoc += 2) {
+ toDocByFromDoc[fromDoc] = 1000 * batch + fromDoc;
+ }
+ columnsByPair.put(pairFieldName, toDocByFromDoc);
+ written.put(pairFieldName, model(toDocByFromDoc));
+ }
+ joinIndex.writeBatch(written);
+ return columnsByPair;
+ }
+
+ /** Every sidecar field belonging to {@code pairFieldName}, across all segments. */
+ private static List<String> fieldsOf(IndexReader sidecar, String pairFieldName) {
+ List<String> found = new ArrayList<>();
+ for (LeafReaderContext leaf : sidecar.leaves()) {
+ for (FieldInfo fieldInfo : leaf.reader().getFieldInfos()) {
+ if (fieldInfo.name.endsWith(pairFieldName)) {
+ found.add(leaf.reader().toString() + ":" + fieldInfo.name);
+ }
+ }
+ }
+ return found;
+ }
+
+ /**
+ * A queued pair's columns have to actually leave the index, and its name has to leave the queue.
+ * Neither used to happen: reaping could only drop a segment all of whose pairs were dead at once,
+ * which a segment carrying many pairs never is, so {@code pendingPairRemovals} only ever grew --
+ * every entry pinning a column that could never be read again, until the heap ran out.
+ */
+ public void testQueuedPairsAreReapedAndDrained() throws Exception {
+ joinIndex.mergePolicy.setCompaction(3, AuxIndexJoinMergePolicy.DEFAULT_MAX_PAIRS_PER_SEGMENT);
+
+ Map<String, int[]> columnsByPair = writeBatchesOfVaryingLength(0, 6);
+ List<String> allPairs = new ArrayList<>(columnsByPair.keySet());
+ List<String> doomed = allPairs.subList(0, 2);
+ for (String pairFieldName : doomed) {
+ joinIndex.mergePolicy.queueForRemoval(pairFieldName);
+ }
+ assertEquals(doomed.size(), joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ // a write is what gives the policy its next look at the index -- reaping rides on merges, it
+ // has no timer of its own
+ columnsByPair.putAll(writeBatchesOfVaryingLength(6, 1));
+ joinIndex.waitForMerges();
+ columnsByPair.putAll(writeBatchesOfVaryingLength(7, 1)); // publishes what just merged
+
+ assertEquals(
+ "every queued pair should have been reaped, so nothing should still be pending",
+ 0,
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ for (String pairFieldName : doomed) {
+ assertEquals(
+ "reaped pair " + pairFieldName + " still has columns in the sidecar",
+ List.of(),
+ fieldsOf(sidecar, pairFieldName));
+ }
+ // and the merge that dropped them left every other column exactly as written
+ for (Map.Entry<String, int[]> pair : columnsByPair.entrySet()) {
+ if (!doomed.contains(pair.getKey())) {
+ assertColumnIntact(sidecar, pair.getKey(), pair.getValue());
+ }
+ }
+ }
+ }
+
+ /**
+ * Compaction cannot be relied on to carry reaping: it needs a quorum of eligible segments, and
+ * {@code isCompactable} passes over exactly the segments most worth purging -- the ones already
+ * wider than {@code maxPairsPerSegment}. Pinned here by making every segment ineligible: only the
+ * single-segment purge path can reclaim anything.
+ */
+ public void testPairsAreReapedEvenWhenCompactionSkipsEverySegment() throws Exception {
+ // maxPairsPerSegment=0 makes isCompactable false for every segment carrying a pair, and the
+ // fold size is beyond anything this test writes, so no DocAlignedMerge can be proposed for
+ // compaction's own sake
+ joinIndex.mergePolicy.setCompaction(1000, 0);
+
+ // two pairs per segment, and only one of them dies: a segment that is merely *partly* dead is
+ // the case the whole-segment drop cannot handle, and the one a real sidecar is always in
+ Map<String, int[]> columnsByPair = new LinkedHashMap<>();
+ String doomed = null;
+ for (int batch = 0; batch < 4; batch++) {
+ Map<String, int[]> written = writeWideBatch(batch, 2);
+ columnsByPair.putAll(written);
+ if (batch == 0) {
+ doomed = written.keySet().iterator().next(); // one pair of a two-pair segment
+ }
+ }
+ // 1 of 2 is well over the purge threshold
+ joinIndex.mergePolicy.queueForRemoval(doomed);
+
+ // a write to give the policy its next look, then one more to publish what merged
+ columnsByPair.putAll(writeBatchesOfVaryingLength(100, 1));
+ joinIndex.waitForMerges();
+ columnsByPair.putAll(writeBatchesOfVaryingLength(101, 1));
+
+ assertEquals(
+ "nothing was compactable, so only the purge path could have reclaimed anything",
+ 0,
+ joinIndex.mergePolicy.alignedMergeCount());
+ assertEquals(
+ "the segment still holds a live pair, so it must not be dropped whole",
+ 0,
+ joinIndex.mergePolicy.droppedSegmentCount());
+ assertTrue(
+ "expected a single-segment purge to rewrite the segment holding the dead column",
+ joinIndex.mergePolicy.purgedSegmentCount() >= 1);
+ assertEquals(0, joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertEquals(List.of(), fieldsOf(sidecar, doomed));
+ for (Map.Entry<String, int[]> pair : columnsByPair.entrySet()) {
+ if (!pair.getKey().equals(doomed)) {
+ assertColumnIntact(sidecar, pair.getKey(), pair.getValue());
+ }
+ }
+ }
+ }
+
+ /**
+ * Runs an already-created {@link Weight} across every leaf, returning the parent ids it matches.
+ */
+ private static Set<String> searchWith(Weight weight, IndexSearcher parents) throws IOException {
+ Set<String> matched = new TreeSet<>();
+ for (LeafReaderContext leaf : parents.getIndexReader().leaves()) {
+ ScorerSupplier supplier = weight.scorerSupplier(leaf);
+ if (supplier == null) {
+ continue;
+ }
+ DocIdSetIterator docs = supplier.get(Long.MAX_VALUE).iterator();
+ for (int doc = docs.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = docs.nextDoc()) {
+ matched.add(parents.storedFields().document(leaf.docBase + doc).get(PARENT_ID));
+ }
+ }
+ return matched;
+ }
+
+ /**
+ * The race that made column-level reaping unsafe, and the reason it was left undone: a query
+ * snapshots the pair columns it found at {@code createWeight} time, and the reaper can drop one
+ * before the scorer supplier resolves it. That used to throw {@code "unable to refresh segment
+ * refs"} and fail the query outright; now the pair is simply rebuilt, since the reap signal is a
+ * heuristic and must be recoverable rather than fatal.
+ */
+ public void testQueryOutlivesTheReapingOfItsOwnColumns() throws Exception {
+ List<String> parentIds = addParentsAndChildren("gen1-", atLeast(9));
+
+ try (IndexReader childrenReader = childrenWriter.getReader();
+ IndexReader parentsReader = parentsWriter.getReader()) {
+ IndexSearcher parents = new IndexSearcher(parentsReader);
+ IndexSearcher children = new IndexSearcher(childrenReader);
+
+ // first pass builds the sidecar, so there are real columns to reap
+ assertEquals(new TreeSet<>(parentIds), searchAllParents(parents, children));
+
+ Query joinQuery =
+ parents.rewrite(
+ joinIndex.newJoinQuery(PARENT_ID_FK, new MatchAllDocsQuery(), children, PARENT_ID));
+ // the weight snapshots which pair columns exist right now...
+ Weight weight = joinQuery.createWeight(parents, ScoreMode.COMPLETE_NO_SCORES, 1f);
+
+ // ...and every one of them is reaped before the scorers resolve it
+ List<String> reaped = new ArrayList<>();
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ for (LeafReaderContext leaf : sidecar.leaves()) {
+ reaped.addAll(JoinIndexUtils.pairFieldNames(leaf.reader().getFieldInfos()));
+ }
+ }
+ assertFalse("the first pass should have built some pair columns", reaped.isEmpty());
+ for (String pairFieldName : reaped) {
+ joinIndex.mergePolicy.queueForRemoval(pairFieldName);
+ }
+ joinIndex.writeBatch(Map.of("trigger:dv0_trigger:dv0", model(new int[] {0})));
+ joinIndex.waitForMerges();
+ joinIndex.writeBatch(Map.of("trigger2:dv0_trigger2:dv0", model(new int[] {0})));
+
+ assertEquals(
+ "the query's columns should be gone by now",
+ 0,
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ // the query, holding references to columns that no longer exist, still answers correctly
+ assertEquals(new TreeSet<>(parentIds), searchWith(weight, parents));
+ }
+ }
+
+ /**
+ * A purge rewrites the whole segment to drop what is dead in it, so it has to reclaim enough
+ * bytes to be worth the trip; until it does, the name simply stays queued. The threshold is in
+ * bytes rather than in share of columns because share is what let the fattest segments -- the
+ * ones holding the bytes -- dilute themselves out of ever being cleaned.
+ */
+ public void testAThinlyDeadSegmentIsNotWorthRewriting() throws Exception {
+ joinIndex.mergePolicy.setCompaction(1000, 0); // nothing compactable, so only purges can act
+
+ Map<String, int[]> columnsByPair = new LinkedHashMap<>(writeWideBatch(0, 20));
+ // between what one column of twenty is worth and what four are: 1/20 must not pay, 4/20 must
+ joinIndex.mergePolicy.setMinReclaimableBytesToPurge(widestSegmentBytes() / 8);
+ String doomed = columnsByPair.keySet().iterator().next();
+ joinIndex.mergePolicy.queueForRemoval(doomed);
+
+ columnsByPair.putAll(writeBatchesOfVaryingLength(100, 1));
+ joinIndex.waitForMerges();
+ columnsByPair.putAll(writeBatchesOfVaryingLength(101, 1));
+
+ assertEquals(
+ "one dead column in twenty does not reclaim enough to pay for rewriting the other nineteen",
+ 0,
+ joinIndex.mergePolicy.purgedSegmentCount());
+ assertEquals(
+ "and the name stays queued, so the next death in that segment can pay for the trip",
+ 1,
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertFalse(
+ "the column is still there, just not yet worth removing",
+ fieldsOf(sidecar, doomed).isEmpty());
+ }
+
+ // now enough of that segment dies to make the rewrite pay
+ List<String> alsoDoomed = new ArrayList<>(columnsByPair.keySet()).subList(1, 4);
+ for (String pairFieldName : alsoDoomed) {
+ joinIndex.mergePolicy.queueForRemoval(pairFieldName);
+ }
+ columnsByPair.putAll(writeBatchesOfVaryingLength(102, 1));
+ joinIndex.waitForMerges();
+ columnsByPair.putAll(writeBatchesOfVaryingLength(103, 1));
+
+ assertTrue(
+ "4 of 20 dead is worth more than the threshold, so the segment should have been purged",
+ joinIndex.mergePolicy.purgedSegmentCount() >= 1);
+ assertEquals(0, joinIndex.mergePolicy.pendingPairRemovalsCount());
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertEquals(List.of(), fieldsOf(sidecar, doomed));
+ for (String pairFieldName : alsoDoomed) {
+ assertEquals(List.of(), fieldsOf(sidecar, pairFieldName));
+ }
+ }
+ }
+
+ /**
+ * The threshold trades heap for I/O, and that trade has to stop once the queue is itself the
+ * danger -- otherwise declining to rewrite is just the old unbounded leak with extra steps.
+ */
+ public void testTheThresholdIsWaivedOnceTheQueueIsDangerous() throws Exception {
+ joinIndex.mergePolicy.setCompaction(1000, 0);
+
+ Map<String, int[]> columnsByPair = new LinkedHashMap<>(writeWideBatch(0, 20));
+ // nothing this segment holds could ever be worth a rewrite on its own
+ joinIndex.mergePolicy.setMinReclaimableBytesToPurge(Long.MAX_VALUE);
+ String doomed = columnsByPair.keySet().iterator().next();
+
+ // a backlog of names for columns long gone, as a stalled reaper would accumulate
+ for (int i = 0; i < 2048; i++) {
+ joinIndex.mergePolicy.queueForRemoval("stale" + i + ":dv0_gone:dv0");
+ }
+ joinIndex.mergePolicy.queueForRemoval(doomed);
+
+ columnsByPair.putAll(writeBatchesOfVaryingLength(100, 1));
+ joinIndex.waitForMerges();
+ columnsByPair.putAll(writeBatchesOfVaryingLength(101, 1));
+
+ assertTrue(
+ "over the high-water mark the threshold must be waived, thin or not",
+ joinIndex.mergePolicy.purgedSegmentCount() >= 1);
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertEquals(List.of(), fieldsOf(sidecar, doomed));
+ for (Map.Entry<String, int[]> pair : columnsByPair.entrySet()) {
+ if (!pair.getKey().equals(doomed)) {
+ assertColumnIntact(sidecar, pair.getKey(), pair.getValue());
+ }
+ }
+ }
+ }
+
+ /**
+ * The reap signal is keyed by directory, so consecutive samples come from different queries that
+ * may hold different searcher generations. An older generation must not be allowed to speak: it
+ * cannot see the segments opened since, and would report every one of them as dead -- queueing
+ * live columns whose queries then have to rebuild them mid-flight.
+ */
+ public void testAnOlderSampleCannotCondemnPairsItCannotSee() throws Exception {
+ addParentsAndChildren("gen1-", 3);
+ try (DirectoryReader parentsGen1 = parentsWriter.getReader();
+ DirectoryReader childrenGen1 = childrenWriter.getReader()) {
+ addParentsAndChildren("gen2-", 3);
+ try (DirectoryReader parentsGen2 = parentsWriter.getReader();
+ DirectoryReader childrenGen2 = childrenWriter.getReader()) {
+ assertTrue(
+ "the second generation must be newer for this test to mean anything",
+ parentsGen2.getVersion() > parentsGen1.getVersion());
+
+ // the newer generation needs both pairs
+ joinIndex.onCreateWeight(
+ Set.of("pairA", "pairB"),
+ PARENT_ID_FK,
+ new IndexSearcher(childrenGen2),
+ PARENT_ID,
+ new IndexSearcher(parentsGen2));
+ assertEquals(0, joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ // an older query samples next and does not need pairB -- because it cannot see it yet
+ joinIndex.onCreateWeight(
+ Set.of("pairA"),
+ PARENT_ID_FK,
+ new IndexSearcher(childrenGen1),
+ PARENT_ID,
+ new IndexSearcher(parentsGen1));
+ assertEquals(
+ "a sample from an older generation must not condemn anything",
+ 0,
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+
+ // whereas the newest generation dropping it is a real death
+ addParentsAndChildren("gen3-", 3);
+ try (DirectoryReader parentsGen3 = parentsWriter.getReader();
+ DirectoryReader childrenGen3 = childrenWriter.getReader()) {
+ joinIndex.onCreateWeight(
+ Set.of("pairA"),
+ PARENT_ID_FK,
+ new IndexSearcher(childrenGen3),
+ PARENT_ID,
+ new IndexSearcher(parentsGen3));
+ assertEquals(
+ "the newest generation no longer needing a pair is what death looks like",
+ Set.of("pairB").size(),
+ joinIndex.mergePolicy.pendingPairRemovalsCount());
+ }
+ }
+ }
+ }
+
+ /**
+ * The failure mode compaction exists for: under a steady stream of pair builds the sidecar used
+ * to grow by one segment per batch forever -- hundreds of segments, each a compound file the
+ * merge policy reopens on every commit, until the JVM runs out of mmap-able address space ("Map
+ * failed" out of MMapDirectory). With compaction the segment count stays bounded instead.
+ */
+ public void testSegmentCountStaysBoundedUnderSteadyWrites() throws Exception {
+ int batches = 60; // with the default fold of 10, one per commit, as a steady load would
+ writeBatchesOfVaryingLength(0, batches);
+ joinIndex.waitForMerges();
+ writeBatchesOfVaryingLength(batches, 1); // a commit publishing whatever just merged
+ joinIndex.waitForMerges();
+
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ assertTrue(
+ "expected compactions to keep the sidecar's segment count well under the "
+ + batches
+ + " batches written, got "
+ + sidecar.leaves().size()
+ + " segments after "
+ + joinIndex.mergePolicy.alignedMergeCount()
+ + " compactions",
+ sidecar.leaves().size() <= batches / 3);
+ }
+ }
+
+ /** The size on disk of the sidecar's biggest segment, which the wide batch above just wrote. */
+ private long widestSegmentBytes() throws IOException {
+ long widest = 0;
+ for (SegmentCommitInfo info : SegmentInfos.readLatestCommit(joinDir)) {
+ widest = Math.max(widest, info.sizeInBytes());
+ }
+ return widest;
+ }
+
+ /** Every pair the sidecar currently stores a column for, across all its segments. */
+ /**
+ * Writing a batch flushes it and nothing more: the flush is what seals it into a segment of its
+ * own -- the invariant a pair column depends on, a sidecar doc number being a from-doc id -- and
+ * the manager's searcher reads the writer's in-memory segments, so the batch answers queries
+ * straight away. Committing it is the timer's job, which is what keeps the fsync off the write
+ * path. Whatever the timer never reached, {@code close()} commits.
+ */
+ public void testABatchIsAnsweredBeforeItIsCommitted() throws Exception {
+ String pairFieldName = "fromSegUncommitted:dv0_toSeg:dv0";
+ int[] toDocByFromDoc = {7, -1, 9};
+ Directory uncommittedDir = newDirectory();
+ try {
+ // an interval no test run will reach, so nothing commits behind these assertions
+ AuxIndexManager sidecar =
+ new AuxIndexManager(
+ uncommittedDir,
+ new AuxIndexJoinConfig().setCommitIntervalMs(TimeUnit.HOURS.toMillis(1)));
+ try {
+ sidecar.writeBatch(Map.of(pairFieldName, model(toDocByFromDoc)));
+ assertEquals(
+ "a flushed batch has to be answerable at once",
+ Set.of(pairFieldName),
+ pairNamesVisibleTo(sidecar));
+ assertEquals(
+ "and must not have cost a commit to get there",
+ Set.of(),
+ committedPairNames(uncommittedDir));
+ } finally {
+ sidecar.close();
+ }
+ assertEquals(
+ "close() has to commit whatever the timer never reached",
+ Set.of(pairFieldName),
+ committedPairNames(uncommittedDir));
+ } finally {
+ uncommittedDir.close();
+ }
+ }
+
+ /** The pairs the sidecar's own searcher can see, committed or merely flushed. */
+ private static Set<String> pairNamesVisibleTo(AuxIndexManager sidecar) throws IOException {
+ IndexSearcher searcher = sidecar.acquire();
+ try {
+ Set<String> names = new TreeSet<>();
+ for (LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
+ names.addAll(JoinIndexUtils.pairFieldNames(leaf.reader().getFieldInfos()));
+ }
+ return names;
+ } finally {
+ sidecar.release(searcher);
+ }
+ }
+
+ /** The pairs a reader opening the directory from scratch can see, i.e. what was committed. */
+ private static Set<String> committedPairNames(Directory dir) throws IOException {
+ Set<String> names = new TreeSet<>();
+ try (DirectoryReader sidecar = DirectoryReader.open(dir)) {
+ for (LeafReaderContext leaf : sidecar.leaves()) {
+ names.addAll(JoinIndexUtils.pairFieldNames(leaf.reader().getFieldInfos()));
+ }
+ } catch (IndexNotFoundException nothingCommittedYet) {
+ return names;
+ }
+ return names;
+ }
+
+ /** A config whose batches land on disk as soon as they are written, as these tests assume. */
+ private static AuxIndexJoinConfig commitEveryBatch() {
+ return new AuxIndexJoinConfig().setCommitIntervalMs(0);
+ }
+
+ private Set<String> storedPairNames() throws IOException {
+ Set<String> names = new TreeSet<>();
+ try (DirectoryReader sidecar = DirectoryReader.open(joinDir)) {
+ for (LeafReaderContext leaf : sidecar.leaves()) {
+ names.addAll(JoinIndexUtils.pairFieldNames(leaf.reader().getFieldInfos()));
+ }
+ }
+ return names;
+ }
+
+ /**
+ * What a restart inherits has to be reclaimable, and the sampled death signal structurally cannot
+ * reclaim it: that signal only fires when a query needs a pair and a later query doesn't, both
+ * within one process, so a column whose from-segment was merged away while Solr was down is
+ * needed by nobody, missed by nobody, and lives forever. Load tests ended with most of the
+ * sidecar's bytes in exactly such columns, inherited from the previous run's directory.
+ *
+ * <p>Here the readers are the evidence instead: whatever side keys they do not offer name
+ * segments that are gone, whether they went during this process or before it started.
+ */
+ public void testColumnsStrandedByARestartAreReaped() throws Exception {
+ List<String> parentIds = addParentsAndChildren("gen1-", atLeast(6));
+ try (IndexReader parentsReader = parentsWriter.getReader();
+ IndexReader childrenReader = childrenWriter.getReader()) {
+ assertEquals(
+ new TreeSet<>(parentIds),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+ Set<String> inherited = storedPairNames();
+ assertFalse("the join should have built columns to inherit", inherited.isEmpty());
+
+ // the restart: a manager opened on the same directory, with no sampling history whatsoever
+ joinIndex.close();
+ joinIndex = new AuxIndexManager(joinDir, commitEveryBatch());
+ joinIndex.mergePolicy.setSweepInterval(0, TimeUnit.NANOSECONDS);
+
+ // and while it was down, every from-segment those columns name was merged away
+ childrenWriter.w.getConfig().setMergePolicy(new TieredMergePolicy());
+ childrenWriter.forceMerge(1);
+ childrenWriter.commit();
+
+ try (IndexReader parentsReader = parentsWriter.getReader();
+ IndexReader childrenReader = childrenWriter.getReader()) {
+ assertEquals(1, childrenReader.leaves().size());
+ // the first query rebuilds against the merged from-segment and commits, which is what puts
+ // the inherited names in front of the policy; the second is the sample that judges them
+ for (int sample = 0; sample < 2; sample++) {
+ assertEquals(
+ "the join must keep answering while its inherited columns are condemned",
+ new TreeSet<>(parentIds),
+ searchAllParents(newSearcher(parentsReader), newSearcher(childrenReader)));
+ }
+ }
+
+ assertTrue(
+ "a restart's own readers show every inherited column's from-segment gone, so the sweep "
+ + "should have condemned all "
+ + inherited.size()
+ + " of them -- the snapshot diff never can, no query here needing them twice -- but it "
+ + "condemned "
+ + joinIndex.mergePolicy.strandedColumnCount(),
+ joinIndex.mergePolicy.strandedColumnCount() >= inherited.size());
+
+ // and queued means reclaimed: the segments holding them are now entirely dead
+ writeBatchesOfVaryingLength(500, 1);
+ joinIndex.waitForMerges();
+ writeBatchesOfVaryingLength(501, 1); // a commit publishing whatever just dropped
+
+ assertTrue(
+ "the inherited columns should have been dropped, reaped=" + reapedOrDropped(),
+ reapedOrDropped() > 0);
+ Set<String> left = storedPairNames();
+ left.retainAll(inherited);
+ assertEquals("no inherited column should still be on disk", Set.of(), left);
+ }
+
+ private int reapedOrDropped() {
+ return joinIndex.mergePolicy.reapedPairCount() + joinIndex.mergePolicy.droppedSegmentCount();
+ }
+}
diff --git a/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestJoinColumnModel.java b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestJoinColumnModel.java
new file mode 100644
index 00000000000..e3191e63ebc
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/search/join/auxindexjoin/TestJoinColumnModel.java
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search.join.auxindexjoin;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.packed.PackedLongValues;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.Edges;
+import org.apache.solr.search.join.auxindexjoin.JoinIndexUtils.JoinColumnModel;
+
+/**
+ * The two {@link JoinColumnModel} layouts have to be indistinguishable through the read API, since
+ * which one a pair got is decided by its density at build time and neither consumer knows or cares.
+ * These tests hold the same from-doc -> to-doc map in both and assert the cursors agree, under the
+ * two access patterns the consumers actually use: {@code JoinColumnDocWriter} walking {@code
+ * nextDoc()} up the batch, and {@code JoinIndexScorerSupplier.LeafJoin} calling {@code
+ * advanceExact} along a forward-only from-doc iterator.
+ */
+public class TestJoinColumnModel extends SolrTestCase {
+
+ public void testSparseAndDenseWalkAlike() throws IOException {
+ for (int iter = 0; iter < 200; iter++) {
+ int maxDoc = TestUtil.nextInt(random(), 1, 500);
+ int[] toDocByFromDoc = randomMap(maxDoc, random().nextFloat());
+ assertWalksAlike(toDocByFromDoc);
+ assertSeeksAlike(toDocByFromDoc);
+ }
+ }
+
+ /** The layouts have to agree at the extremes too: nothing matched, and everything matched. */
+ public void testDegenerateColumns() throws IOException {
+ int maxDoc = TestUtil.nextInt(random(), 1, 100);
+ int[] empty = new int[maxDoc];
+ Arrays.fill(empty, -1);
+ assertWalksAlike(empty);
+ assertSeeksAlike(empty);
+
+ int[] full = new int[maxDoc];
+ for (int fromDoc = 0; fromDoc < maxDoc; fromDoc++) {
+ full[fromDoc] = maxDoc - 1 - fromDoc;
+ }
+ assertWalksAlike(full);
+ assertSeeksAlike(full);
+ }
+
+ /** A tombstone costs no per-doc storage, but still stands for its whole from-segment. */
+ public void testTombstoneHoldsNothingButItsWidth() throws IOException {
+ JoinColumnModel tombstone = JoinColumnModel.tombstone(5_000_000);
+ assertEquals(5_000_000, tombstone.maxDoc());
+ assertEquals(0, tombstone.edges().toCount());
+ assertTrue(tombstone.isSparse());
+ // it costs a pair of empty page arrays, whatever from-segment it stands for
+ assertEquals(JoinColumnModel.tombstone(10).ramBytesUsed(), tombstone.ramBytesUsed());
+ assertTrue(String.valueOf(tombstone.ramBytesUsed()), tombstone.ramBytesUsed() < 1024);
+ assertEquals(DocIdSetIterator.NO_MORE_DOCS, tombstone.toDocByFromDoc().nextDoc());
+ assertFalse(tombstone.toDocByFromDoc().advanceExact(random().nextInt(5_000_000)));
+ }
+
+ /** The layout is chosen on how many values each remembers, so it flips at exactly 50% density. */
+ public void testLayoutFlipsAtHalfDensity() {
+ assertTrue(JoinColumnModel.sparseIsSmaller(0, 1_000));
+ assertTrue(JoinColumnModel.sparseIsSmaller(499, 1_000));
+ assertFalse(JoinColumnModel.sparseIsSmaller(500, 1_000));
+ assertFalse(JoinColumnModel.sparseIsSmaller(1_000, 1_000));
+ // the shape that motivated the split: a five-million-doc from-segment matching 2,125 docs
+ assertTrue(JoinColumnModel.sparseIsSmaller(2_125, 5_031_945));
+ assertFalse(JoinColumnModel.sparseIsSmaller(5_031_876, 5_031_945));
+ }
+
+ /**
+ * Whatever the column -- or the from-side map, or the to-side inversion, which page at the same
+ * size -- its widest possible page stays under half of a 1MB G1 region, so none of them is ever
+ * allocated humongous, which is what paging them is for. Neither {@link PackedLongValues} nor
+ * {@code PagedMutable} lays down a page wider than the page size at the full 64 bits.
+ */
+ public void testNoPageCanBeHumongous() {
+ assertTrue(
+ "page size " + JoinIndexUtils.PAGE_SIZE + " can exceed a G1 humongous threshold",
+ (long) JoinIndexUtils.PAGE_SIZE * Long.BYTES < 512 * 1024);
+ }
+
+ /**
+ * A dense column of the width that crashed production costs a fraction of the flat {@code int[]}
+ * it replaces, because a to-doc is stored at the bits it needs rather than at 32.
+ */
+ public void testWideDenseColumnPacksBelowItsIntArray() {
+ int maxDoc = 2_000_000;
+ JoinColumnModel.Builder column = JoinColumnModel.builder(maxDoc, false);
+ for (int fromDoc = 0; fromDoc < maxDoc; fromDoc++) {
+ column.add(fromDoc, maxDoc - 1 - fromDoc);
+ }
+ JoinColumnModel dense =
+ column.build(new Edges(new int[] {0, maxDoc - 1}, new int[] {0, maxDoc - 1}, maxDoc));
+ assertFalse(dense.isSparse());
+ long asIntArray = (long) maxDoc * Integer.BYTES;
+ assertTrue(
+ "packed " + dense.ramBytesUsed() + " against " + asIntArray + " bytes flat",
+ dense.ramBytesUsed() < asIntArray * 3 / 4);
+ }
+
+ /** A sparse column really is sized to its matches, not to its from-segment. */
+ public void testSparseColumnCostsWhatItHolds() {
+ int maxDoc = 1_000_000;
+ int[] toDocByFromDoc = new int[maxDoc];
+ Arrays.fill(toDocByFromDoc, -1);
+ for (int fromDoc = 0; fromDoc < maxDoc; fromDoc += 1_000) {
+ toDocByFromDoc[fromDoc] = fromDoc;
+ }
+ JoinColumnModel sparse = sparse(toDocByFromDoc);
+ assertTrue(sparse.isSparse());
+ assertEquals(maxDoc, sparse.maxDoc());
+ assertTrue(
+ "a 0.1% dense column should cost far less than its dense twin: " + sparse.ramBytesUsed(),
+ sparse.ramBytesUsed() * 100 < dense(toDocByFromDoc).ramBytesUsed());
+ }
+
+ /** Walks both layouts end to end with {@code nextDoc()}, as the batch writer does. */
+ private void assertWalksAlike(int[] toDocByFromDoc) throws IOException {
+ List<String> denseWalk = walk(dense(toDocByFromDoc).toDocByFromDoc());
+ List<String> sparseWalk = walk(sparse(toDocByFromDoc).toDocByFromDoc());
+ assertEquals(sparseWalk, denseWalk);
+ assertEquals(expectedWalk(toDocByFromDoc), sparseWalk);
+ }
+
+ /**
+ * Probes both layouts at an ascending sample of from-docs with {@code advanceExact}, as a drain
+ * does along its from-doc iterator.
+ */
+ private void assertSeeksAlike(int[] toDocByFromDoc) throws IOException {
+ SortedNumericDocValues denseCursor = dense(toDocByFromDoc).toDocByFromDoc();
+ SortedNumericDocValues sparseCursor = sparse(toDocByFromDoc).toDocByFromDoc();
+ for (int fromDoc = 0;
+ fromDoc < toDocByFromDoc.length;
+ fromDoc += TestUtil.nextInt(random(), 1, 8)) {
+ boolean denseHit = denseCursor.advanceExact(fromDoc);
+ assertEquals("at from-doc " + fromDoc, denseHit, sparseCursor.advanceExact(fromDoc));
+ assertEquals("at from-doc " + fromDoc, toDocByFromDoc[fromDoc] >= 0, denseHit);
+ if (denseHit) {
+ assertEquals(toDocByFromDoc[fromDoc], denseCursor.nextValue());
+ assertEquals(toDocByFromDoc[fromDoc], sparseCursor.nextValue());
+ }
+ assertEquals(fromDoc, sparseCursor.docID());
+ }
+ }
+
+ private static List<String> walk(SortedNumericDocValues cursor) throws IOException {
+ List<String> walked = new ArrayList<>();
+ for (int doc = cursor.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = cursor.nextDoc()) {
+ assertEquals(1, cursor.docValueCount());
+ walked.add(doc + "->" + cursor.nextValue());
+ }
+ return walked;
+ }
+
+ private static List<String> expectedWalk(int[] toDocByFromDoc) {
+ List<String> expected = new ArrayList<>();
+ for (int fromDoc = 0; fromDoc < toDocByFromDoc.length; fromDoc++) {
+ if (toDocByFromDoc[fromDoc] >= 0) {
+ expected.add(fromDoc + "->" + toDocByFromDoc[fromDoc]);
+ }
+ }
+ return expected;
+ }
+
+ /** A from-doc -> to-doc map where each from-doc matches with probability {@code density}. */
+ private static int[] randomMap(int maxDoc, float density) {
+ int[] toDocByFromDoc = new int[maxDoc];
+ for (int fromDoc = 0; fromDoc < maxDoc; fromDoc++) {
+ toDocByFromDoc[fromDoc] = random().nextFloat() < density ? random().nextInt(maxDoc) : -1;
+ }
+ return toDocByFromDoc;
+ }
+
+ private static JoinColumnModel dense(int[] toDocByFromDoc) {
+ return JoinColumnModel.dense(toDocByFromDoc, edgesOf(toDocByFromDoc));
+ }
+
+ /** The same map as {@link #dense}, laid out as matches only. */
+ private static JoinColumnModel sparse(int[] toDocByFromDoc) {
+ return JoinColumnModel.sparse(toDocByFromDoc, edgesOf(toDocByFromDoc));
+ }
+
+ private static Edges edgesOf(int[] toDocByFromDoc) {
+ int minFromDoc = -1;
+ int maxFromDoc = -1;
+ int minToDoc = -1;
+ int maxToDoc = -1;
+ int toCount = 0;
+ for (int fromDoc = 0; fromDoc < toDocByFromDoc.length; fromDoc++) {
+ int toDoc = toDocByFromDoc[fromDoc];
+ if (toDoc < 0) {
+ continue;
+ }
+ if (toCount == 0) {
+ minFromDoc = fromDoc;
+ minToDoc = toDoc;
+ maxToDoc = toDoc;
+ }
+ maxFromDoc = fromDoc;
+ minToDoc = Math.min(minToDoc, toDoc);
+ maxToDoc = Math.max(maxToDoc, toDoc);
+ toCount++;
+ }
+ return new Edges(new int[] {minFromDoc, maxFromDoc}, new int[] {minToDoc, maxToDoc}, toCount);
+ }
+}
diff --git a/solr/solr-ref-guide/modules/query-guide/pages/auxindexjoin-query-parser.adoc b/solr/solr-ref-guide/modules/query-guide/pages/auxindexjoin-query-parser.adoc
new file mode 100644
index 00000000000..fce6ba53cad
--- /dev/null
+++ b/solr/solr-ref-guide/modules/query-guide/pages/auxindexjoin-query-parser.adoc
@@ -0,0 +1,185 @@
+= Auxiliary Index Join Query Parser
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License
+
+`auxIndexJoin` (which stands for Auxiliary Index Join) Query Parser is similar to xref:join-query-parser.adoc[], but uses a lazily written sidecar index for faster joins.
+
+Instead of using Lucene's join utilities, the parser matches documents through a dedicated join index that is maintained alongside the core's main index.
+The join index is populated lazily: if the inner (`from`-side) query doesn't hit a certain segment, the join index isn't written for it; and if `{!auxIndexJoin}` is intersected (`AND`/`+`) with a query that doesn't hit a certain segment on the outer (`to`) side, the corresponding join index columns aren't written either.
+
+Like the Join query parser, Solr runs a subquery (the `v` parameter), gathers the values that matching documents have in a `from` field, and returns documents where those values are contained in a `to` field.
+
+For example:
+
+[source,text]
+----
+q={!auxIndexJoin from=manu_id_s to=id}title:ipod
+----
+
+== Parameters
+
+This query parser takes the following parameters:
+
+`from`::
++
+[%autowidth,frame=none]
+|===
+s|Required |Default: none
+|===
++
+The "foreign key" field name, collected while enumerating the subordinate query.
+So far, it should be single value string field with docValues enabled.
+
+`to`::
++
+[%autowidth,frame=none]
+|===
+s|Required |Default: none
+|===
++
+The "primary key" field name looked up in the local core's index.
+So far, it should be single value string field with docValues enabled.
+
+`fromIndex`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: processing core
+|===
++
+The name of the core to run the "from" query (`v` parameter) on and where "from" values are gathered.
+If this parameter is not defined, it defaults to the processing core.
+Cross-core joins are the primary use case for `auxIndexJoin`, so this is useful when the "from" values live in a different core on the same node.
+
+== Configuration
+
+`auxIndexJoin` query parser *must be* registered in `solrconfig.xml` as a `<queryParser>`.
+A sidecar join index is opened per core when the core loads, in a directory under the core's `dataDir`, and closed when the core closes.
+
+`dir`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `aux-index-join`
+|===
++
+The init parameter for the directory holding the sidecar join index.
+Resolved relative to the core's `dataDir` unless an absolute path is given.
+
+`singleFieldPerSegment`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `false`
+|===
++
+If `true`, each pair column is flushed into its own sidecar segment.
+Otherwise, every pair column built in the same round is batched into a single segment (the default), trading a longer sweep for less storage overhead.
+
+`blockingRefresh`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `true`
+|===
++
+If `true`, writing a batch of pair columns blocks until the sidecar's searcher manager is refreshed past it, so the freshly built pairs are immediately visible to the caller that triggered the build.
+
+`useFromSideThreads`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `true`
+|===
++
+If `true` (the default), searching the from-side segments and loading foreigh key columns is parallelized across the executor threads described in <<segment-level-parallelism>>.
+Note it's actual only if <<segment-level-parallelism>> is enabled and treated as `false` otherwise.
+If `false`, the from-side segments are loaded sequentially on the calling thread, bounding the build to a single thread.
+
+`sweepSamplingInterval`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: `60`
+|===
++
+How often (in seconds) the dead-pair reaper samples searcher state while the sidecar index is being read.
+Calls arriving sooner than this interval after the last accepted sample are skipped.
+Pass a non-positive value to sample on every call.
+
+[source,xml]
+----
+<queryParser name="auxIndexJoin" class="org.apache.solr.search.join.AuxIndexJoinQParserPlugin">
+ <str name="dir">aijoin</str>
+ <bool name="singleFieldPerSegment">false</bool>
+ <bool name="blockingRefresh">true</bool>
+ <bool name="useFromSideThreads">true</bool>
+ <long name="sweepSamplingInterval">60</long>
+</queryParser>
+----
+
+[[segment-level-parallelism]]
+== Segment-level Parallelism
+
+The algorithm utilizes multiple threads on _both_ sides if multiple threads are available. So, set xref:configuration-guide:configuring-solr-xml.adoc#indexSearcherExecutorThreads[indexSearcherExecutorThreads] to `-1` or `>0` in the `solr.xml` file. Note the emphasis on _both_ above: using the xref:common-query-parameters.adoc#multithreaded-parameter[`multiThreaded`] request parameter confines the inner (`from`-side) query to a single thread, limiting performance.
+
+[[sizing-the-from-side]]
+== Sizing the From Side and the JVM
+
+What the sidecar costs is driven by how large the `from`-side segments are, not by how many documents a join actually matches.
+For each (`from`-segment, `to`-segment) pair it builds, the parser holds several structures sized by the `from` segment's `maxDoc`: the set of matching `from` documents, the map from each `from` document to its field value, and the batch written to the sidecar, which spans one document per `from` document however few of them the pair matched.
+On a `from` segment of several million documents each of those runs to tens of megabytes, so the two settings below are worth applying before a large join is put under sustained load.
+
+NOTE: Both are starting points drawn from benchmarking a single relation, not tuned defaults. Like the rest of this parser they are provisional; see <<Disclaimer>>.
+
+=== Cap the From-Side Segment Size
+
+Put a ceiling on how large the `from` core's segments may grow, in that core's `solrconfig.xml`:
+
+[source,xml]
+----
+<indexConfig>
+ <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
+ <double name="maxMergedSegmentMB">200.0</double>
+ </mergePolicyFactory>
+</indexConfig>
+----
+
+This is a ceiling on future merges rather than a reduction: it prevents segments growing past the limit, but doesn't split segments that are already larger.
+
+`TieredMergePolicy` has no document-count setting, so this bounds documents only indirectly, through bytes.
+The document count a given `maxMergedSegmentMB` works out to depends on the average document size, and is worth re-checking whenever the schema changes.
+
+Resist capping too aggressively.
+The parser maintains one pair column per (`from`-segment, `to`-segment) combination, so halving the `from` segment size doubles the number of `from` segments, and with it the number of pair columns to build, persist, and later reclaim.
+
+=== Raise the G1 Region Size
+
+[source,bash]
+----
+GC_TUNE="-XX:G1HeapRegionSize=8m"
+----
+
+G1 treats any single allocation larger than half a region as "`humongous`", placing it in its own run of regions that ordinary young collections don't reclaim.
+At G1's default 1 MB region that threshold is 512 KB, which the per-segment structures described above cross on segments of a few million documents -- a bitset over a `from` segment passes it at roughly 4.2 million documents.
+An 8 MB region moves the threshold to 4 MB, which brings those allocations back under normal collection.
+
+See xref:deployment-guide:jvm-settings.adoc[] for where JVM settings live and how to set them.
+
+== Disclaimer
+
+`auxIndexJoin` is completely experimental, which is why it requires explicit configuration. So, far it's evalueated within single relation (from-to index pair) only, but it should support many ones in principle. It's parameters/usage is more subject to change with less backwards compatibility concern.
diff --git a/solr/solr-ref-guide/modules/query-guide/pages/common-query-parameters.adoc b/solr/solr-ref-guide/modules/query-guide/pages/common-query-parameters.adoc
index 7acd8024fee..7c8f1765246 100644
--- a/solr/solr-ref-guide/modules/query-guide/pages/common-query-parameters.adoc
+++ b/solr/solr-ref-guide/modules/query-guide/pages/common-query-parameters.adoc
@@ -454,7 +454,9 @@ Similar to using <<timeAllowed Parameter,the `timeAllowed` Parameter>>, when ear
|===
This parameter set to `true` or `false` controls if Solr may use more than one thread to satisfy the request.
-A `true` value presently allows the IndexSearcher to search across Lucene's segments in parallel, and the xref:configuration-guide:configuring-solr-xml.adoc#indexSearcherExecutorThreads[indexSearcherExecutorThreads] value can be customised in the `solr.xml` file.
+A `true` value presently allows the IndexSearcher to search across Lucene's segments in parallel,
+and the xref:configuration-guide:configuring-solr-xml.adoc#indexSearcherExecutorThreads[indexSearcherExecutorThreads]
+value can be customized in the `solr.xml` file.
This parameter is ignored in the presence of `&segmentsTerminateEarly=true` (future work may enable it).
This is a new parameter and is considered experimental and subject to change or removal in subsequent releases.
Please share your feedback and experiences with it on our mailing lists.
diff --git a/solr/solr-ref-guide/modules/query-guide/pages/other-parsers.adoc b/solr/solr-ref-guide/modules/query-guide/pages/other-parsers.adoc
index f1966f0b767..2be81f40302 100644
--- a/solr/solr-ref-guide/modules/query-guide/pages/other-parsers.adoc
+++ b/solr/solr-ref-guide/modules/query-guide/pages/other-parsers.adoc
@@ -22,6 +22,13 @@ This section details the other parsers, and gives examples for how they might be
Many of these parsers are expressed the same way as xref:local-params.adoc[].
+== Auxiliary Index Join Query Parser
+
+It's similar to
+<<Join Query Parser,Join Query Parser>>, but uses a lazily written sidecar index for faster joins.
+
+Details of this query parser are in the section xref:auxindexjoin-query-parser.adoc[].
+
== Block Join Query Parsers
The Block Join query parsers are used with nested documents to query for parents and/or children.