You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "haohuaijin (via GitHub)" <gi...@apache.org> on 2026/09/13 12:45:20 UTC

[I] Join dynamic filter pushdown through FilterExec with a projection can discard matching rows with duplicate column names [datafusion]

haohuaijin opened a new issue, #25262:
URL: https://github.com/apache/datafusion/issues/25262

   ### Describe the bug
   
   Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a `FilterExec` that carries an embedded projection over an input with duplicate column names (here `a.id` and `b.id` from a nested join).
   
   With the same data and SQL, disabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row; enabling it returns zero rows, without an error.
   
   ### To Reproduce
   
   Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/` Parquet paths must not already exist.
   
   ```sql
   SET datafusion.optimizer.join_reordering = false;
   
   COPY (SELECT 'a1' AS id, 'x1' AS ty)
   TO '/tmp/df_dynamic_filter_filterprojection_a.parquet'
   STORED AS PARQUET;
   
   COPY (SELECT 'x1' AS id)
   TO '/tmp/df_dynamic_filter_filterprojection_b.parquet'
   STORED AS PARQUET;
   
   CREATE EXTERNAL TABLE ta
   STORED AS PARQUET
   LOCATION '/tmp/df_dynamic_filter_filterprojection_a.parquet';
   
   CREATE EXTERNAL TABLE tb
   STORED AS PARQUET
   LOCATION '/tmp/df_dynamic_filter_filterprojection_b.parquet';
   
   -- Returns one row: (x1, a1, x1).
   SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
   
   SELECT s.id AS sid, a.id AS aid, b.id AS bid
   FROM tb s
   JOIN (
       SELECT a.id, b.id
       FROM ta a LEFT JOIN tb b ON a.ty = b.id
       WHERE (a.ty || '!') IS DISTINCT FROM b.id
   )
   ON s.id = b.id;
   
   -- Incorrectly returns zero rows.
   SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
   
   SELECT s.id AS sid, a.id AS aid, b.id AS bid
   FROM tb s
   JOIN (
       SELECT a.id, b.id
       FROM ta a LEFT JOIN tb b ON a.ty = b.id
       WHERE (a.ty || '!') IS DISTINCT FROM b.id
   )
   ON s.id = b.id;
   ```
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   `FilterExec` resolves pushed-down filter columns by name instead of through its embedded projection, so the filter on `b.id` is applied to `a.id`. Found while working on #25244; the fix is included in #25259.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Join dynamic filter pushdown through `FilterExec` with a projection can discard matching rows with duplicate column names [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #25262: Join dynamic filter pushdown through `FilterExec` with a projection can discard matching rows with duplicate column names
URL: https://github.com/apache/datafusion/issues/25262


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org