You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "erratic-pattern (via GitHub)" <gi...@apache.org> on 2026/02/19 19:27:55 UTC
[I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]
erratic-pattern opened a new issue, #20437:
URL: https://github.com/apache/datafusion/issues/20437
# Panic in HashJoin with dictionary-encoded column in multi-column join key
## Describe the bug
When executing a hash join with multiple join keys where one column is dictionary-encoded with fewer unique values than rows, DataFusion panics with:
```
InvalidArgumentError("Incorrect array length for StructArray field \"c1\", expected N got M")
```
## To Reproduce
```sql
-- Small table with dictionary-encoded region (2 rows, 1 unique value)
CREATE TABLE small AS
SELECT id, arrow_cast(region, 'Dictionary(Int32, Utf8)') as region
FROM (VALUES (1, 'west'), (2, 'west')) AS t(id, region);
CREATE TABLE large AS
SELECT id, region, value
FROM (VALUES (1, 'west', 100), (2, 'west', 200), (3, 'east', 300)) AS t(id, region, value);
-- Multi-column join triggers panic
SELECT s.id, s.region, l.value
FROM small s
JOIN large l ON s.id = l.id AND s.region = l.region;
```
## Expected behavior
Query returns 2 rows:
```
+----+--------+-------+
| id | region | value |
+----+--------+-------+
| 1 | west | 100 |
| 2 | west | 200 |
+----+--------+-------+
```
## Actual behavior
Panic:
```
thread 'main' panicked at arrow-array/src/array/struct_array.rs:91:46:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Incorrect array length for StructArray field \"c1\", expected 3 got 2")
```
## Root cause
In [`flatten_dictionary_array`](https://github.com/apache/datafusion/blob/52.1.0/datafusion/physical-plan/src/joins/hash_join/inlist_builder.rs#L37-L45) introduced by #18393:
```rust
fn flatten_dictionary_array(array: &ArrayRef) -> ArrayRef {
downcast_dictionary_array! {
array => {
flatten_dictionary_array(array.values()) // BUG: returns only unique values
}
_ => Arc::clone(array)
}
}
```
The function calls `array.values()` which returns the dictionary's unique values array (length = number of unique values), not the expanded array (length = number of rows).
When building a `StructArray` for multi-column join keys, this causes a length mismatch between dictionary columns (incorrectly shortened) and non-dictionary columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20437:
URL: https://github.com/apache/datafusion/issues/20437#issuecomment-3940828380
This is also technically a regression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]
Posted by "Tim-53 (via GitHub)" <gi...@apache.org>.
Tim-53 commented on issue #20437:
URL: https://github.com/apache/datafusion/issues/20437#issuecomment-3930783505
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #20437: Panic in HashJoin with dictionary-encoded column in multi-column join key
URL: https://github.com/apache/datafusion/issues/20437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org