You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "berkaysynnada (via GitHub)" <gi...@apache.org> on 2024/08/08 10:28:23 UTC
[I] Configurable null_equals_null flag [datafusion]
berkaysynnada opened a new issue, #11883:
URL: https://github.com/apache/datafusion/issues/11883
### Is your feature request related to a problem or challenge?
Currently the `null_equals_null` flag in DF joins is hard-coded to false across multiple places in the code. It is not configurable either from a query or session configuration. We need to make this flag configurable to allow more flexibility in how joins handle null values.
### Describe the solution you'd like
I am not sure which alternative is more useful and expected, and I couldn't find any examples. If we select it as a session config, users will need to change the setting for each preference change. On the other hand, if it is provided within the query, users must explicitly specify it for each non-default case.
### Describe alternatives you've considered
Perhaps both solutions can coexist.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Configurable null_equals_null flag [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on issue #11883:
URL: https://github.com/apache/datafusion/issues/11883#issuecomment-2283657039
That makes sense. If such usages exist, there's no need to expose a new flag. It would be great if we could handle these two versions and link them to our internal join flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Configurable null_equals_null flag [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #11883:
URL: https://github.com/apache/datafusion/issues/11883#issuecomment-2282747848
Just some thoughts: maybe it's better not to expose this flag, but set it while planning in case of queries like
```
with data as (
select 1 as f
union all
select 2 as f
union all
select null as f
)
select * from data t1
join data t2
on t1.f is not distinct from t2.f; -- it (probably) may also be cool to support <=> operator here
```
right now it's planned as cross (technically) join with filter, but in fact this query may be a hash join with null equals null. And specifying join conditions like this (via `is not distinct` or spaceship operator) potentially more convenient to control join behaviour (one query statements instead of `set datafusion.xxx = true` + query statement)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Configurable null_equals_null flag [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #11883:
URL: https://github.com/apache/datafusion/issues/11883#issuecomment-2282755292
And in the end this field, probably, is not needed (in its current form at least) as it won't help to handle queries like
```sql
JOIN ... ON left.x = right.x AND left.y <=> right.y
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org