You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "erratic-pattern (via GitHub)" <gi...@apache.org> on 2026/03/13 21:20:56 UTC
[I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
erratic-pattern opened a new issue, #20937:
URL: https://github.com/apache/datafusion/issues/20937
### Describe the bug
When a `VALUES` clause is joined against a Parquet table with `Dictionary(Int32, Utf8)` columns and `pushdown_filters = true`, the query fails with:
```
Parquet error: External: Compute error: Error evaluating filter predicate:
ArrowError(InvalidArgumentError("Can't compare arrays of different types"), Some(""))
```
The dynamic filter pushdown from the `HashJoinExec` creates an `InListExpr` that is pushed into the Parquet scan. [`ArrayStaticFilter::contains()`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L90) unwraps the needle (the Dictionary array from the Parquet batch) to its plain Utf8 values via [`downcast_dictionary_array!`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L102-L108), but does not unwrap `in_array` (the stored InList values, which are also Dictionary from the type-coerced join build side). This results in a [`make_comparator(Utf8, Dictionary)`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L116) call, which [`arrow_ord::ord::build_compare`](https://docs.rs/arrow-ord/57.3.0/src/arrow_ord/ord
.rs.html#475-490) does not support for mixed Dictionary/non-Dictionary types.
### To Reproduce
Using the DataFusion CLI (`datafusion-cli`):
```sql
-- Enable row-level filter pushdown (required to trigger the bug)
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.execution.parquet.reorder_filters = true;
-- Create a Parquet file with Dictionary-encoded string columns
COPY (
SELECT
arrow_cast(chr(65 + (row_num % 26)), 'Dictionary(Int32, Utf8)') as tag1,
row_num * 1.0 as value
FROM (SELECT unnest(range(0, 10000)) as row_num)
) TO '/tmp/dict_filter_bug.parquet';
-- This query fails
SELECT t.tag1, t.value
FROM '/tmp/dict_filter_bug.parquet' t
JOIN (VALUES ('A'), ('B')) AS v(c1)
ON t.tag1 = v.c1;
```
### Expected behavior
The query should return matching rows (tag1 = 'A' or 'B').
### Actual behavior
```
Parquet error: External: Compute error: Error evaluating filter predicate:
ArrowError(InvalidArgumentError("Can't compare arrays of different types"), Some(""))
```
### Root cause
The error path through the code:
1. The VALUES clause produces `Utf8` strings. Type coercion casts them to `Dictionary(Int32, Utf8)` to match the Parquet column type for the join key.
2. The `HashJoinExec` builds the dynamic filter InList from the cast build-side arrays → `in_array` is `Dictionary(Int32, Utf8)`.
3. [`instantiate_static_filter()`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L157-L159) receives the Dictionary array and falls through to `ArrayStaticFilter::try_new()`, which stores `in_array` as-is.
4. [`InListExpr::try_new_from_array()`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L647) is called from [`create_membership_predicate()`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs#L124-L128) in the dynamic filter pushdown code.
5. At runtime, `ArrayStaticFilter::contains(v)` is called with a Dictionary array from the Parquet batch.
6. [`downcast_dictionary_array!`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L102-L108) matches and recursively calls `self.contains(v.values())` with the unwrapped Utf8 values.
7. In the recursive call, `v` is now `Utf8` but `self.in_array` is still `Dictionary(Int32, Utf8)`.
8. [`make_comparator(Utf8, Dictionary)`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L116) fails because `arrow_ord::ord::build_compare` has no arm for mixed Dictionary/non-Dictionary types.
### Possible fix
The most straightforward fix would be to normalize `in_array` in [`ArrayStaticFilter::try_new()`](https://github.com/apache/arrow-datafusion/blob/d09ff92def5ad5f033e75bbfda1fd9d33a4c93c9/datafusion/physical-expr/src/expressions/in_list.rs#L187) — if the input is a Dictionary, unwrap it to its values array before storing. This is a one-time cost at filter construction, and ensures that after `contains()` unwraps the needle Dictionary, both sides are the same plain type and the existing arrow comparison kernels work correctly.
### Additional context
- This only manifests when `pushdown_filters = true` (defaults to `false`), because the row-level filter evaluation inside the Parquet reader is what triggers `ArrayStaticFilter::contains()` with Dictionary arrays. With `pushdown_filters = false`, the InList filter is only used for statistics/bloom filter pruning (which works correctly), and the actual row filtering happens in the HashJoin.
- The bug applies to single-column equi-joins where the VALUES side is cast to Dictionary by type coercion and the dynamic filter InList path is taken (small build side).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067476555
> Before that fix, `flatten_dictionary_array()` in `inlist_builder.rs` converted Dictionary arrays to plain Utf8 before building the InList, so `in_array` was always Utf8 and the `make_comparator(Utf8, Utf8)` call in `ArrayStaticFilter::contains()` worked.
I think @adriangb even predicted this in
- https://github.com/apache/datafusion/pull/20505#issuecomment-3998523892
I suggest we get the reproducer in and we can figure out the right fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "erratic-pattern (via GitHub)" <gi...@apache.org>.
erratic-pattern commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067561994
Reproducer: https://github.com/apache/datafusion/pull/20960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067844712
I just verified that the reproducer fails on main and branch-53
I updated my table in https://github.com/apache/datafusion/issues/20937#issuecomment-4067056646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "erratic-pattern (via GitHub)" <gi...@apache.org>.
erratic-pattern commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067466143
That lines up with what I've investigated. The regression was introduced by:
- #20505
Before that fix, `flatten_dictionary_array()` in `inlist_builder.rs` converted Dictionary arrays to plain Utf8 before building the InList, so `in_array` was always Utf8 and the `make_comparator(Utf8, Utf8)` call in `ArrayStaticFilter::contains()` worked.
After the fix removed flattening, Dictionary arrays are preserved in the InList. When `contains()` unwraps the probe-side Dictionary to Utf8 via `downcast_dictionary_array!`, the stored `in_array` is still Dictionary, causing the type mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067403009
It seems like this was introduce in 52.3.0 -- I am investigating
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067712755
I just verified this happens on `branch-52` but **NOT** `branch-53` or datafusion-main 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067056646
I ran this reproducer using datafusion-cli
| version | status |
|--------|--------|
| datafusion-cli 52.0.0 | pass |
| datafusion-cli 52.1.0 | pass |
| datafusion-cli 52.2.0 | pass |
| datafusion-cli 52.3.0 | (in progress) |
| datafusion-cli branch-52 (to be 52.4.0) | (in progress) |
| datafusion-cli branch-53 (to be 53.0.0) | (in progress) |
| datafusion-cli main | (in progress) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns [datafusion]
Posted by "erratic-pattern (via GitHub)" <gi...@apache.org>.
erratic-pattern commented on issue #20937:
URL: https://github.com/apache/datafusion/issues/20937#issuecomment-4067735015
Interesting that it passes on branch-53 and main. branch-53 still has `flatten_dictionary_array` (it doesn't have #20505), which explains why it passes there.
But main has #20505 (flatten removed) and the same `ArrayStaticFilter::contains()` code as branch-52. Could you double-check the main result? The code paths look identical to branch-52 from my reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org