[jira] [Updated] (SOLR-18571) Query Insights: opt-in top-N expensive queries per node

["Amrit Sarkar (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amrit Sarkar updated SOLR-18571:
--------------------------------
    Attachment: query-insights-design.pdf

> Query Insights: opt-in top-N expensive queries per node
> -------------------------------------------------------
>
>                 Key: SOLR-18571
>                 URL: https://issues.apache.org/jira/browse/SOLR-18571
>             Project: Solr
>          Issue Type: New Feature
>          Components: metrics, search, v2 API
>            Reporter: Amrit Sarkar
>            Priority: Major
>         Attachments: query-insights-design.pdf
>
>
> Umbrella for Query Insights: an opt-in module that keeps the most expensive searches on each node in memory and returns them through one v2 API across the cluster. It implements the idea in SOLR-11994.
> Discussed on dev@: [thread|https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd]. No objections came in on the thread.
> *Shape*
> * Off by default. Turned on by adding {{query-insights}} to {{SOLR_MODULES}} and setting {{SOLR_QUERYINSIGHTS_ENABLED=true}}. Without both, nothing loads and each search pays one {{isEmpty()}} check.
> * Records only on the coordinator. Top-N by latency and by CPU over clock-aligned windows; the current and previous window are returned. Defaults: N=10 (max 100), 5 minute windows.
> * Each record has collection, handler, params (redactable with {{params.hidden}}, capped at 8 KB), hits, shard count, user, trace id, latency, CPU, and the costly features the query used (re-ranking, LTR, knn, grouping, faceting, paramsets).
> * {{GET /api/node/query-insights/top-queries?type=latency|cpu&nodes=all}}, needs {{metrics-read}}. Optional {{collection}} and {{traceId}} filters. Nodes that don't answer are listed in {{failedNodes}}.
> * Defaults from system properties; runtime changes through {{ext.queryInsights.*}} cluster properties.
> *Plan* (one PR each, each reviewable and mergeable on its own)
> # SOLR-11994: {{SearchRequestListener}} hook in core. No behavior change when nothing is registered.
> # The {{query-insights}} module: per-node top-N and the node-local API.
> # Cluster-wide view: {{nodes=all}} merge with {{failedNodes}}.
> # {{collection}} and {{traceId}} filters, runtime settings through cluster properties.
> # Ref guide page.
> *Defaults for the open questions from the thread* (happy to change any of these in review)
> * Module plus a small core listener, rather than all in core.
> * {{metrics-read}} permission, with {{params.hidden}} redaction.
> * {{SearchRequestListener}} marked {{@lucene.experimental}}.
> * {{/export}} is recorded; {{/get}} is not.
> *Later, separate sub-tasks*: stored history as a Time Routed Alias with {{autoDeleteAge}} (to be lined up with [UBI PR #2452|https://github.com/apache/solr/pull/2452]), grouping similar queries by shape, memory as a metric, an Admin UI page.
> Full design is attached as {{query-insights-design.pdf}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org