You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "prudhvigodithi (via GitHub)" <gi...@apache.org> on 2026/07/24 22:23:28 UTC

[PR] Support intra-segment search in the query profiler with per-slice and per-partition breakdowns [lucene]

prudhvigodithi opened a new pull request, #16417:
URL: https://github.com/apache/lucene/pull/16417

   ### Description
   
   Building on intra-segment search concurrency (#13542) and concurrent-search profiling (#14413), this PR attributes query-profiler timings per **slice** and per **partition**, resolving the slice-identification problem left open in #14413's review. #14413 keys leaf breakdowns by `Thread`, so when one thread runs several slices sequentially their timings merge into one bucket and for intra-segment search it cannot attribute timing to the doc-id-range partitions a segment is split into.
   
     **Change**
     - Reliably identifies slices via a counter assigned at the once-per-slice `search(LeafReaderContextPartition[])` seam, independent of thread.
     - Keys leaf timings per partition `(sliceId, segmentOrd, minDocId, maxDocId)`, set at `searchLeaf(...)`; both overrides delegate to `super` (no duplicated search logic).
     - Single code path — no concurrent/non-concurrent class split; non-concurrent search is the single-slice degenerate case (one slice covering whole-segment partitions).
     - Per-slice `threadId`/`threadName`/`startTime` show how the search parallelized, `Thread` object replaced by serializable id+name
   
   
     ### Sample output
   
     Verified locally by adding a temporary `toString()` (not included in this PR) and serializing `getProfileResult()` for a `BooleanQuery(+foo:bar foo:baz)` over a single segment split into two partitions `[0,100)` / `[100,200)` on a 2-thread executor. 
    ```json
    [
       {
         "type": "BooleanQuery",
         "description": "+foo:bar foo:baz",
         "breakdown": { "create_weight": 1634167, "create_weight_count": 1 },
         "slices": [
           {
             "slice_id": 0,
             "thread_id": 22,
             "thread_name": "search-worker-caller",
             "partitions": [
               { "segment_ord": 0, "doc_range": [0, 100],
                 "breakdown": { "build_scorer": 5185916, "next_doc_count": 101, "score_count": 100, "advance_count": 0 } }
             ]
           },
           {
             "slice_id": 1,
             "thread_id": 23,
             "thread_name": "search-worker-pool-3-thread-1",
             "partitions": [
               { "segment_ord": 0, "doc_range": [100, 200],
                 "breakdown": { "build_scorer": 5183792, "next_doc_count": 100, "score_count": 100, "advance_count": 1 } }
             ]
           }
         ],
         "children": [
           {
             "type": "TermQuery", "description": "foo:bar",
             "breakdown": { "create_weight": 903875, "create_weight_count": 1 },
             "slices": [
               { "slice_id": 0, "thread_id": 22, "thread_name": "search-worker-caller",
                 "partitions": [ { "segment_ord": 0, "doc_range": [0, 100],
                   "breakdown": { "build_scorer": 1026292, "next_doc_count": 101, "score_count": 100 } } ] },
               { "slice_id": 1, "thread_id": 23, "thread_name": "search-worker-pool-3-thread-1",
                 "partitions": [ { "segment_ord": 0, "doc_range": [100, 200],
                   "breakdown": { "build_scorer": 1008001, "next_doc_count": 100, "score_count": 100 } } ] }
             ],
             "children": []
           },
           {
             "type": "TermQuery", "description": "foo:baz",
             "breakdown": { "create_weight": 52459, "create_weight_count": 1 },
             "slices": [
               { "slice_id": 0, "thread_id": 22, "thread_name": "search-worker-caller",
                 "partitions": [ { "segment_ord": 0, "doc_range": [0, 100],
                   "breakdown": { "build_scorer": 959, "next_doc_count": 0, "score_count": 0 } } ] },
               { "slice_id": 1, "thread_id": 23, "thread_name": "search-worker-pool-3-thread-1",
                 "partitions": [ { "segment_ord": 0, "doc_range": [100, 200],
                   "breakdown": { "build_scorer": 708, "next_doc_count": 0, "score_count": 0 } } ] }
             ],
             "children": []
           }
         ]
       }
     ]
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Support intra-segment search in the query profiler with per-slice and per-partition breakdowns [lucene]

Posted by "prudhvigodithi (via GitHub)" <gi...@apache.org>.
prudhvigodithi commented on PR #16417:
URL: https://github.com/apache/lucene/pull/16417#issuecomment-5096842140

   Just noticed for the segments which does not partition (whole segments) the end value shows the `getMaxDocId` which is not the real segment size. Pushed a commit to fix this, example
   
     | segment | before           | after   | real docs |
     |---------|------------------|---------|------------------------------|
     | 0       | [0, 2147483647]  | [0, 30] | 30                           |
     | 1       | [0, 2147483647]  | [0, 20] | 20                           |
     | 2       | [0, 2147483647]  | [0, 10] | 10                           |
   
   The split partition segments works as expected


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Support intra-segment search in the query profiler with per-slice and per-partition breakdowns [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16417:
URL: https://github.com/apache/lucene/pull/16417#issuecomment-5287956990

   This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org