[I] Remove vendored PushDownLeafProjections workaround after DataFusion upgrade [sedona-db]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot opened a new issue, #1232:
URL: https://github.com/apache/sedona-db/issues/1232

   ## Problem
   
   SedonaDB currently depends on DataFusion 54.1.0, which contains a logical optimizer correctness bug in `PushDownLeafProjections` when nested-field access is applied to a struct produced by `Unnest`.
   
   For a plan shaped like:
   
   ```text
   Projection: get_field(dump, "geom")
     Unnest: dump
       Projection: ST_Dump(geometry) AS dump
   ```
   
   the optimizer can push `get_field(dump, "geom")` below `Unnest`. At that point `dump` is still `List<Struct<...>>`, rather than the post-unnest `Struct<...>`, and optimization fails with:
   
   ```text
   Optimizer rule 'push_down_leaf_projections' failed
   caused by
   Execution error: Cannot access field at argument 1: type List(Struct(...))
   is not Struct, Map, or Null
   ```
   
   This is a plan-correctness bug, not a security issue.
   
   ## Upstream status
   
   The bug was reported and fixed upstream:
   
   - apache/datafusion#22615 describes `PushDownLeafProjections` crossing `Unnest`, including the same-named pre/post-unnest column problem.
   - apache/datafusion#22620 fixes it by treating `Unnest` as a semantic barrier for leaf-projection pushdown.
   - apache/datafusion#22783 subsequently makes the `LogicalPlan::Unnest` expression/rebuild contracts consistent while retaining the semantic barrier.
   
   The reduced reproducer fails with DataFusion 54.1.0 and succeeds on DataFusion `main` at `e1942b144`.
   
   ## Temporary SedonaDB workaround
   
   Until SedonaDB upgrades to a DataFusion release containing apache/datafusion#22620, `sedona-query-planner` replaces DataFusion's optimizer rule named `push_down_leaf_projections` with a compatibility rule that prevents leaf projection pushdown across `Unnest`.
   
   Regression coverage includes both a DataFusion-only `List<Struct>` plan and SedonaDB's `ST_Dump → unnest → geom` workflow.
   
   ## Action item
   
   - [ ] After upgrading DataFusion to a release containing apache/datafusion#22620, remove the vendored `PushDownLeafProjections` compatibility rule and its registration, while retaining appropriate regression coverage against the upstream rule.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org