[PR] OAK-search-mongot: cut per-query round trips for the next POC build [jackrabbit-oak]

["meticulous-dft (via GitHub)" <gi...@apache.org>]

meticulous-dft opened a new pull request, #3170:
URL: https://github.com/apache/jackrabbit-oak/pull/3170

   ## Summary
   
   This PR replaces #3158, #3159 and #3160 with one change set, so the next AEM test build for the POC needs a single deployment. Adobe's first benchmark ran from Azure West Europe against an Atlas cluster in Azure `eastus2`, about 90 ms per round trip. At that distance, the number of sequential round trips per query dominates latency. mongot itself served the benchmark's search commands in 30 to 40 ms.
   
   | Commit | Author | Was | Change |
   |---|---|---|---|
   | Let the connector bundle resolve in OSGi | Ming He | #3158 | Embed only `oak-search`, and deploy the MongoDB driver bundles in the OSGi test runtime, so `OSGiIT.bundleStates` passes |
   | Drop per-query search index lookups and redundant sorts | Ming He | #3159 | Check the search index only when a `$search` returns no hits; count without the sort and projection; no sort for `jcr:score` descending |
   | Count matches inside mongot through the search metadata (2 commits) | Oren Ovadia | #3160 | For a bare `[$search, $project]` pipeline, take the exact count from `$searchMeta` with `count: {type: "total"}` instead of a `$count` pipeline |
   | Cache the index statistics Oak reads while planning | Ming He | new | Serve `numDocs()` and `getDocCountFor()` from a cache, as the Elasticsearch index does |
   
   The statistics cache addresses the largest remaining cost. Oak's planner reads `numDocs()` and `getDocCountFor(property)` for every query, and each read was a `countDocuments` against the index collection: 2 to 3 extra round trips before the query even started. The counts now come from a cache with a 60-second background refresh and a 10-minute expiry (`oak.mongot.statsRefreshSeconds`, `oak.mongot.statsExpireSeconds`), the same defaults as the Elasticsearch index. The cache belongs to the index node, and the tracker replaces that node whenever the index status or definition changes, so each indexing cycle that writes to the index starts from fresh counts.
   
   **Round trips per query.** These are aggregate plus `getMore` commands the server received, for six AEM-shaped query shapes over 2,004 assets. Each cell shows reading all rows, then in brackets reading all rows plus the exact size.
   
   | Query | Branch tip (#3143) | This PR |
   |---|---|---|
   | property-exact (1 hit) | 4 (6) | 1 (3) |
   | prefix | 5 (7) | 2 (4) |
   | sorted-by-date | 6 (11) | 3 (5) |
   | tag-filtered | 6 (8) | 3 (5) |
   | format-facet | 6 (10) | 3 (6) |
   | fulltext-phrase | 6 (10) | 3 (5) |
   
   What remains is the query, one `getMore` per 1,000 results and, for an exact size, the count query. The other extra round trip comes from Oak itself: calling `getSize()` and then iterating runs the query twice.
   
   **Notes on the combined count path**
   - `$searchMeta` applies only when nothing follows the search. Queries with a path, type or property `$match` after the search, which includes every AEM benchmark shape today, keep the `$count` fallback until filters move into the search stage.
   - `count: {type: "total"}` is exact; the threshold applies only to `lowerBound`.
   - Against a missing search index, `$searchMeta` returns a count of 0 rather than failing. The main query runs first and fails on a missing index, so a size request never reports 0 for one.
   
   **Reviewer focus**
   - `MongotIndex.aggregateCursor`: the search index check now runs only when a `$search` has no first hit, or while retrying an index that is not yet queryable.
   - `MongotIndex.countMatches`: the metadata count comes first, then the leaner `$count` fallback.
   - `MongotIndexStatistics`: the cache keys (the whole collection, or the encoded property field) and its refresh and expiry settings.
   - `oak-search-mongot/pom.xml` and `oak-it-osgi/test-bundles.xml`: the driver is no longer embedded, and a runtime must provide MongoDB driver 5.x bundles, 5.4 or later.
   
   ## Testing
   
   - Added `MongotIndexStatisticsTest`, which counts the server's `$group` stages (`countDocuments`) to show that repeated planning reads don't reach MongoDB; it fails against the uncached statistics.
   - Kept the tests from #3159 and #3160, which count the stages the server runs. The #3159 tests were checked to fail against the code they replace; for Oren's `$searchMeta` test we rely on his report in #3160. The dropped-index test guards behavior that must stay the same.
   - Manually ran `oak-it-osgi`'s integration tests, and all 14 pass, including `OSGiIT.bundleStates`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OAK-search-mongot: cut per-query round trips for the next POC build [jackrabbit-oak]

["Joscorbe (via GitHub)" <gi...@apache.org>]

Joscorbe merged PR #3170:
URL: https://github.com/apache/jackrabbit-oak/pull/3170


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org