You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jacksonrnewhouse (via GitHub)" <gi...@apache.org> on 2024/02/16 20:28:34 UTC
[I] Joins on Structs fail at runtime [arrow-datafusion]
jacksonrnewhouse opened a new issue, #9254:
URL: https://github.com/apache/arrow-datafusion/issues/9254
### Describe the bug
If you attempt to join two tables on a struct field, the query will plan it successfully, albeit with the struct equality in a the `filter`, rather than in the `on` vector. However, when it runs it fails with "Invalid comparison operation". In particular, it triggers this error from arrow-rs: https://github.com/apache/arrow-rs/blob/db811083669df66992008c9409b743a2e365adb0/arrow-ord/src/cmp.rs#L202.
### To Reproduce
I wrote a failing test that just does a self join at https://github.com/apache/arrow-datafusion/compare/35.0.0...ArroyoSystems:arrow-datafusion:bug_report/struct_join_fails_at_execution. The failure message is
```
thread 'user_defined::user_defined_aggregates::test_struct_join' panicked at datafusion/core/tests/user_defined/user_defined_aggregates.rs:172:60:
called `Result::unwrap()` on an `Err` value: Execution("Fail to build join indices in NestedLoopJoinExec, error:Arrow error: Invalid argument error: Invalid comparison operation: Struct([Field { name: \"value\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"time\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) == Struct([Field { name: \"value\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"time\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])")
```
### Expected behavior
Either the join should fail at planning, reporting a clear error that joins on structs are not supported or, preferably, datafusion should support joins on two structs of the same type.
### Additional context
This comes up with Arroyo where we want to join on time windows, e.g. sliding and tumbling windows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Joins on Structs fail at runtime [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9254: Joins on Structs fail at runtime
URL: https://github.com/apache/datafusion/issues/9254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Joins on Structs fail at runtime [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9254:
URL: https://github.com/apache/datafusion/issues/9254#issuecomment-3746122903
Works as of DatFusion 52
```sql
> create table t1 as values ({foo: 'bar'});
0 row(s) fetched.
Elapsed 0.035 seconds.
> create table t2 as values ({foo: 'bar'});
0 row(s) fetched.
Elapsed 0.001 seconds.
> select * from t1 JOIN t2 ON (t1.column1 = t2.column1);
+------------+------------+
| column1 | column1 |
+------------+------------+
| {foo: bar} | {foo: bar} |
+------------+------------+
1 row(s) fetched.
Elapsed 0.012 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Joins on Structs fail at runtime [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9254:
URL: https://github.com/apache/arrow-datafusion/issues/9254#issuecomment-1975197868
👍 thanks -- looks like @jayzhan211 has been hard at work upstream in arrow-rs trying to get the appropriate support put in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Joins on Structs fail at runtime [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9254:
URL: https://github.com/apache/datafusion/issues/9254#issuecomment-3746126725
(the join is done in the the plan, btw)
```sql
> explain select * from t1 JOIN t2 ON (t1.column1 = t2.column1);
+---------------+------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------+
| physical_plan | ┌───────────────────────────┐ |
| | │ HashJoinExec │ |
| | │ -------------------- │ |
| | │ on: ├──────────────┐ |
| | │ (column1 = column1) │ │ |
| | └─────────────┬─────────────┘ │ |
| | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
| | │ DataSourceExec ││ DataSourceExec │ |
| | │ -------------------- ││ -------------------- │ |
| | │ bytes: 296 ││ bytes: 296 │ |
| | │ format: memory ││ format: memory │ |
| | │ rows: 1 ││ rows: 1 │ |
| | └───────────────────────────┘└───────────────────────────┘ |
| | |
+---------------+------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.004 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org