You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/05/06 02:57:40 UTC

[I] Preserve recursive CTE declared schema when aligning physical children [datafusion]

kosiew opened a new issue, #22034:
URL: https://github.com/apache/datafusion/issues/22034

   ## Summary
   
   #21912 introduced a SQL behavior change: an existing recursive CTE SLT had to be amended from `0 AS level` to `SUM(0) AS level` to pass.
   
   A follow-up fix is required because recursive CTE physical alignment must preserve the CTE's declared/static schema, including nullability, rather than widening the recursive CTE output schema and forcing SQL rewrites.
   
   ## Background
   
   Issue #21910 is about preventing recursive CTE batches from leaking recursive-term field names when the recursive term is planned independently from the anchor/static term.
   
   #21912 adds a reusable execution-layer schema helper and wires recursive CTE execution to avoid ad hoc schema rebinding. During review/testing, another problem appeared:
   
   - The anchor/static term can declare a non-null field, e.g. `0 AS level`.
   - The recursive term can compute the same column through an expression that is nullable, e.g. `MIN(rs.level) + 1`.
   - If `RecursiveQueryExec` declares a widened schema using `static_nullable || recursive_nullable`, downstream planning sees a physical schema that no longer matches the logical/static CTE schema.
   - Changing SQL from `0 AS level` to `SUM(0) AS level` makes the logical anchor nullable too, but only hides the mismatch.
   
   ## Evidence
   
   The SQL behavior change is visible in the amended SLT test:
   
   ```diff
   diff --git a/datafusion/sqllogictest/test_files/cte.slt b/datafusion/sqllogictest/test_files/cte.slt
   index e9c1c0245..d13e0d4f0 100644
   --- a/datafusion/sqllogictest/test_files/cte.slt
   +++ b/datafusion/sqllogictest/test_files/cte.slt
   @@ -699,7 +699,7 @@ WITH RECURSIVE region_sales AS (
        SELECT
            s.salesperson_id AS salesperson_id,
            SUM(s.sale_amount) AS amount,
   -        0 as level
   +        SUM(0) as level
        FROM
            sales s
        GROUP BY
   ```
   
   Reverting this SLT query change from:
   
   ```sql
   SUM(0) AS level
   ```
   
   to the original:
   
   ```sql
   0 AS level
   ```
   
   fails with:
   
   ```text
   Physical input schema should be the same as the one converted from logical input schema.
   Differences:
     - field nullability at index 2 [level]: (physical) true vs (logical) false.
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Recursive CTE Nullability Handling Should Preserve Logical Schema Without Requiring SQL Rewrites [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed issue #22034: Recursive CTE Nullability Handling Should Preserve Logical Schema Without Requiring SQL Rewrites
URL: https://github.com/apache/datafusion/issues/22034


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org