You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2026/09/06 17:05:25 UTC

[I] Adapative Query Runner (AQE) PoC [pinot]

wirybeaver opened a new issue, #19485:
URL: https://github.com/apache/pinot/issues/19485

   ## Goal
   
   Build an incremental Adaptive Query Runner PoC for Pinot's multi-stage query engine. The work first proves a runtime-informed execution decision at an existing pipeline breaker, then adds the execution seams required to adapt an undispatched downstream stage after its producer completes.
   
   ## Roadmap
   
   - [ ] **AQE-PR-1 — Minimal executable AQE rule:** #19470
     - Skip probe-side leaf execution when a completed dynamic-broadcast SEMI JOIN build side is empty.
     - Demonstrates a real runtime decision without changing the stage graph or worker routing.
   
   - [ ] **AQE-PR-2 — Materialized exchange:** #19471
     - Add opt-in, consumer-independent materialized HASH partitions.
     - Publish exact output handles and row/byte statistics after producer completion.
   
   - [ ] **AQE-PR-3 — Staged dispatch:** #19472
     - Dispatch stages in dependency order.
     - Hold downstream stages behind all-worker completion barriers while preserving the original assignments.
   
   - [ ] **AQE-PR-4 — Late-bound routing:** #19483
     - Validate the completed producer's exact partition-handle coverage.
     - Bind materialized inputs to downstream workers immediately before serialization and dispatch.
   
   ## Execution model
   
   ```text
   producer stage completes
             |
             v
   materialized partitions + runtime statistics
             |
             v
   completion barrier releases downstream stage
             |
             v
   late-bind partition handles to consumer workers
             |
             v
   dispatch the ready consumer stage
   ```
   
   ## PoC boundaries
   
   - Opt-in only; existing execution remains the default.
   - No join reordering.
   - No dynamic stage insertion or removal.
   - No cancellation or replacement of running stages.
   - No retries or stage-attempt fencing in this PoC.
   - AQE-PR-2 through AQE-PR-4 retain the planned worker counts; they establish the seams needed for a later deterministic `M -> K` partition-coalescing rule.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Adapative Query Runner (AQE) PoC [pinot]

Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on issue #19485:
URL: https://github.com/apache/pinot/issues/19485#issuecomment-5985575362

   ### End-to-end PoC evidence — AQE-PR-5
   
   Locally verified for [wirybeaver/pinot#73](https://github.com/wirybeaver/pinot/pull/73), commit `54cd91b914`, using a **two-server test fixture with real broker/server gRPC and materialized partition reads**—not only rule/helper unit tests.
   
   | Test target | Observed result |
   |---|---|
   | Nonempty GROUP BY: adapt consumer parallelism after producer completion. | `M -> 1` consumers; every original partition handle accounted for; result rows and schema match the nonadaptive baseline. No missing worker reports. |
   | Empty GROUP BY: coalesce empty partitions. | One consumer, the same empty result/schema as baseline, and complete worker-report coverage. |
   | Physical replanning through a second, **test-only** rule. | The replacement broker plan actually executes; the removed consumer is never dispatched; unused materialized output is cleaned up. |
   
   **Result: 5 real-RPC cases passed, 0 failures**, including two fixed-parallelism staged-dispatch regression cases.
   
   [Test source: QueryRunnerTest](https://github.com/wirybeaver/pinot/blob/54cd91b914e073c3d18bd1c41b37052594dbc5bf/pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/queries/QueryRunnerTest.java#L102-L209).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org