You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "erratic-pattern (via GitHub)" <gi...@apache.org> on 2026/02/19 19:27:55 UTC

[I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]

erratic-pattern opened a new issue, #20437:
URL: https://github.com/apache/datafusion/issues/20437

   # Panic in HashJoin with dictionary-encoded column in multi-column join key
   
   ## Describe the bug
   
   When executing a hash join with multiple join keys where one column is dictionary-encoded with fewer unique values than rows, DataFusion panics with:
   
   ```
   InvalidArgumentError("Incorrect array length for StructArray field \"c1\", expected N got M")
   ```
   
   ## To Reproduce
   
   ```sql
   -- Small table with dictionary-encoded region (2 rows, 1 unique value)
   CREATE TABLE small AS
   SELECT id, arrow_cast(region, 'Dictionary(Int32, Utf8)') as region
   FROM (VALUES (1, 'west'), (2, 'west')) AS t(id, region);
   
   CREATE TABLE large AS
   SELECT id, region, value
   FROM (VALUES (1, 'west', 100), (2, 'west', 200), (3, 'east', 300)) AS t(id, region, value);
   
   -- Multi-column join triggers panic
   SELECT s.id, s.region, l.value
   FROM small s
   JOIN large l ON s.id = l.id AND s.region = l.region;
   ```
   
   ## Expected behavior
   
   Query returns 2 rows:
   ```
   +----+--------+-------+
   | id | region | value |
   +----+--------+-------+
   | 1  | west   | 100   |
   | 2  | west   | 200   |
   +----+--------+-------+
   ```
   
   ## Actual behavior
   
   Panic:
   ```
   thread 'main' panicked at arrow-array/src/array/struct_array.rs:91:46:
   called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Incorrect array length for StructArray field \"c1\", expected 3 got 2")
   ```
   
   ## Root cause
   
   In [`flatten_dictionary_array`](https://github.com/apache/datafusion/blob/52.1.0/datafusion/physical-plan/src/joins/hash_join/inlist_builder.rs#L37-L45) introduced by #18393:
   
   ```rust
   fn flatten_dictionary_array(array: &ArrayRef) -> ArrayRef {
       downcast_dictionary_array! {
           array => {
               flatten_dictionary_array(array.values())  // BUG: returns only unique values
           }
           _ => Arc::clone(array)
       }
   }
   ```
   
   The function calls `array.values()` which returns the dictionary's unique values array (length = number of unique values), not the expanded array (length = number of rows).
   
   When building a `StructArray` for multi-column join keys, this causes a length mismatch between dictionary columns (incorrectly shortened) and non-dictionary columns.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20437:
URL: https://github.com/apache/datafusion/issues/20437#issuecomment-3940828380

   This is also technically a regression


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]

Posted by "Tim-53 (via GitHub)" <gi...@apache.org>.
Tim-53 commented on issue #20437:
URL: https://github.com/apache/datafusion/issues/20437#issuecomment-3930783505

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Panic in HashJoin with dictionary-encoded column in multi-column join key [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #20437: Panic in HashJoin with dictionary-encoded column in multi-column join key
URL: https://github.com/apache/datafusion/issues/20437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org