You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/02/19 21:54:35 UTC

[I] Substrait consumer incorrectly parses references to outer query fields [datafusion]

neilconway opened a new issue, #20438:
URL: https://github.com/apache/datafusion/issues/20438

   ### Describe the bug
   
   In a substrait plan containing correlated subqueries, references to outer query fields are parsed incorrectly: we only look at the schema for the inner/subquery schema, which produced incorrect column names and types.
   
   ### To Reproduce
   
   For example, consider TPC-H Q21, which is parsed as:
   
   ```
           Projection: SUPPLIER.S_NAME, count(Int64(1)) AS NUMWAIT
              Limit: skip=0, fetch=100
                Sort: count(Int64(1)) DESC NULLS FIRST, SUPPLIER.S_NAME ASC NULLS LAST
                  Aggregate: groupBy=[[SUPPLIER.S_NAME]], aggr=[[count(Int64(1))]]
                    Projection: SUPPLIER.S_NAME
                      Filter: SUPPLIER.S_SUPPKEY = LINEITEM.L_SUPPKEY AND ORDERS.O_ORDERKEY = LINEITEM.L_ORDERKEY AND ORDERS.O_ORDERSTATUS = Utf8("F") AND LINEITEM.L_RECEIPTDATE > LINEITEM.L_COMMITDATE AND EXISTS (<subquery>) AND NOT EXISTS (<subquery>) AND SUPPLIER.S_NATIONKEY = NATION.N_NATIONKEY AND NATION.N_NAME = U\
    tf8("SAUDI ARABIA")
                        Subquery:
                          Filter: LINEITEM.L_ORDERKEY = LINEITEM.L_TAX AND LINEITEM.L_SUPPKEY != LINEITEM.L_LINESTATUS
                            TableScan: LINEITEM
                        Subquery:
                          Filter: LINEITEM.L_ORDERKEY = LINEITEM.L_TAX AND LINEITEM.L_SUPPKEY != LINEITEM.L_LINESTATUS AND LINEITEM.L_RECEIPTDATE > LINEITEM.L_COMMITDATE
                            TableScan: LINEITEM
                        Cross Join:
                          Cross Join:
                            Cross Join:
                              TableScan: SUPPLIER
                              TableScan: LINEITEM
                            TableScan: ORDERS
                          TableScan: NATION
   ```
   
   Note that in the subquery, the filter has the clause `LINEITEM.L_SUPPKEY != LINEITEM.L_LINESTATUS`. This is not what Q21 contains; and in fact the types of those two columns (Int64 and Utf8) are not even compatible, although we don't currently reject this.
   
   ### Expected behavior
   
   Parse references to outer query fields correctly.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Substrait consumer incorrectly parses references to outer query fields [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #20438:
URL: https://github.com/apache/datafusion/issues/20438#issuecomment-3930346491

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org