[jira] [Updated] (DRILL-8557) Velox-based native execution

["Letian Jiang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Letian Jiang updated DRILL-8557:
--------------------------------
    Summary: Velox-based native execution  (was: Velox-based native execution for Apache Drill)

> Velox-based native execution
> ----------------------------
>
>                 Key: DRILL-8557
>                 URL: https://issues.apache.org/jira/browse/DRILL-8557
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Flow
>            Reporter: Letian Jiang
>            Priority: Major
>         Attachments: 01-homogeneous.png, 02-heterogeneous.png, 03-sidecar.png, 04-execution-model.png
>
>
> h2. Motivation
> Explore Velox-based native execution to improve CPU efficiency and performance for analytical workloads while retaining Drill’s SQL planning, distributed scheduling, and storage plugin ecosystem.
> Velox already supports native execution in [Presto Native|https://github.com/prestodb/presto/blob/master/presto-native-execution/README.md] and [Spark SQL through Gluten|https://github.com/apache/gluten/blob/main/docs/get-started/Velox.md]. [Gluten for Flink|https://github.com/apache/gluten/blob/main/gluten-flink/docs/Flink.md] also provides an experimental integration. These projects offer useful references for bringing native execution to Drill.
> h2. Proposed design
> Use a *homogeneous Drillbit architecture*: each Java Drillbit can embed the same C++ NativeEngine.
> * *Node discovery:* Each Drillbit retains one ZooKeeper registration and advertises optional native execution endpoints. The Foreman submits native fragments directly through RPC.
> * *Coordination and transport:* Java retains planning, query coordination, and the complete root fragment. Non-root fragments execute in C++ and exchange data through native RPC; results return to the Java root through Drill Data RPC.
> * *Deployment:* The same architecture supports single-node and multi-node deployments. The Foreman’s Drillbit may also execute native non-root fragments.
> !01-homogeneous.png|width=900!
> * *Execution model:* Each non-root minor fragment maps to one Velox Task, which runs pipelines and drivers using shared CPU and scan/I/O pools.
> * *Scan paths:* Existing Java plugin readers are reused through JNI in the current JVM. Optional native readers retain the plugin’s metadata and split planning while keeping non-root scanning, computation, and exchange in C++, avoiding Java/native FFI on that data processing path.
> !04-execution-model.png|width=900!
> h2. Dynamic schema boundary
> The experimental native path supports initial runtime schema discovery, but requires the schema to remain stable after native plan binding. Subsequent schema changes currently fail the query. The existing Java execution path remains available for workloads requiring dynamic schema handling.
> h2. Current status
> I am currently working on a PoC to validate this design, including Java plugin reuse, optional native scans, fragment execution and exchange, correctness, cancellation, and performance comparisons with Java execution.
> h2. Alternative designs
> *Heterogeneous Drillbits:* Java Drillbits handle query coordination and root fragments; separate native Drillbits execute non-root fragments. Both node types register independently and communicate through RPC. Maintaining two node types increases deployment and lifecycle management complexity.
> !02-heterogeneous.png|width=900!
> *Native executor sidecar:* Each Java Drillbit runs alongside a separate C++ executor, sharing one logical node identity and communicating through RPC. This adds process management overhead; reusing Java readers through JNI also requires a JVM inside the sidecar, since JNI cannot cross process boundaries.
> !03-sidecar.png|width=900!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)