[I] Rebase vendored Substrait proto from v0.23.0 to 0.98.0 [gluten]

["nielspardon (via GitHub)" <gi...@apache.org>]

nielspardon opened a new issue, #12597:
URL: https://github.com/apache/gluten/issues/12597

   ### Background
   
   Gluten vendors its Substrait `.proto` files (`gluten-substrait/src/main/resources/substrait/proto/substrait/`) pinned near Substrait **v0.23.0** with local patches, while upstream Substrait is now at **0.98.0**. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is **no wire-compatibility constraint** and the rebase is a pure code migration.
   
   ### Goal
   
   Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.
   
   ### Approach
   
   Land as small, independently-green PRs, one per message family (each keeps `protoc` + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.
   
   ### Increments
   
   - [ ] **0** — Remove unused vendored proto files + dead proto-based type derivation + deprecated `Expression.Enum`
   - [ ] **1** — `plan.proto` / `extensions.proto` → 0.98 (URI→URN, `AdvancedExtension.optimization` repeated)
   - [ ] **2** — Type system: temporal types → `PrecisionTimestamp*`; drop `Type.Nothing`
   - [ ] **3** — `CrossRel` → `NestedLoopJoinRel` (+ join-type enum remap)
   - [ ] **4** — Windowing: `WindowRel` → `ConsistentPartitionWindowRel`; `WindowType` → `BoundsType`
   - [ ] **5** — `FetchRel` / `AggregateRel.Grouping` / `ExpandRel`
   - [ ] **6** — `TopNRel` / `WriteRel`
   - [ ] **7** — `ReadRel` (relocate `stream_kafka`; text-options redesign; `VirtualTable`)
   - [ ] **8** — `Rel` oneof reconciliation to exact 0.98 layout
   
   Related: upstream consolidation work is tracked under the `apache-gluten` label in `substrait-io/substrait`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [I] Rebase vendored Substrait proto from v0.23.0 to 0.98.0 [gluten]

["nielspardon (via GitHub)" <gi...@apache.org>]

nielspardon commented on issue #12597:
URL: https://github.com/apache/gluten/issues/12597#issuecomment-5126993781

   **Update — `Type.Nothing` direction changed (Increment 2)**
   
   The Increment 2 checklist item previously read "drop `Type.Nothing`". That's been corrected to **relocate** (kind `33`→`40`, kept).
   
   Rationale: in the Substrait community sync on 2026-07-29 the PMCs decided upstream should add a dedicated core `null` type rather than have producers avoid a bottom type. That proposal is up as [substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147) (`Type.Null` at `kind` field `40`). Increment 2 deliberately relocates Gluten's `Nothing` fork to field 40 to match, so once #1147 is merged and released upstream the fork converges by a **pure message rename** `Nothing`→`Null` — no field renumbering and no changes to the Velox/ClickHouse consumer mappings. Gluten keeps `Type.Nothing` as-is until then.
   
   Net effect: Increment 2 is semantically neutral for `Nothing` (relocate only), and the former "drop" step becomes a future rename gated on the upstream `Type.Null` release.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org