[PR] fix: serialize optimized logical plan to fix subquery support [datafusion-ballista]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new pull request, #1586:
URL: https://github.com/apache/datafusion-ballista/pull/1586

   ## Summary
   
   Fixes TPC-H Q2 (and any query using subqueries) failing in the Python client with:
   
   ```
   Proto serialization error: Expr::ScalarSubquery(_) | Expr::InSubquery(_) | Expr::Exists { .. } | Expr::OuterReferenceColumn not supported
   ```
   
   **Root cause:** `_to_internal_df()` was calling `logical_plan().to_proto()`, which serializes the *unoptimized* plan. Subquery expressions (`ScalarSubquery`, `InSubquery`, `Exists`, `OuterReferenceColumn`) are not supported by DataFusion's proto serializer, and they're still present in the unoptimized plan.
   
   The Rust client is unaffected because DataFusion's query optimizer runs *before* `BallistaQueryPlanner::create_physical_plan()` is called — `DecorrelatePredicateSubquery` converts subqueries into joins first, so the plan handed to the serializer contains no subquery nodes.
   
   **Fix:** one line — use `optimized_logical_plan().to_proto()` so the Python client serializes the same optimizer output that the Rust path sends to the scheduler.
   
   Fixes #1581
   
   ## Test plan
   
   - [ ] TPC-H Q2 runs successfully via Python client
   - [ ] All 22 TPC-H queries pass via Python client
   - [ ] Existing Python tests pass
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] fix: [Python] serialize optimized logical plan to fix subquery support [datafusion-ballista]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove merged PR #1586:
URL: https://github.com/apache/datafusion-ballista/pull/1586


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org