You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ctsk (via GitHub)" <gi...@apache.org> on 2025/04/20 16:43:12 UTC
[I] Remove nulls in joins during hash table lookup [datafusion]
ctsk opened a new issue, #15784:
URL: https://github.com/apache/datafusion/issues/15784
### Is your feature request related to a problem or challenge?
The join implementation currently removes nulls during equality checking - I believe it would be faster / more efficient if we did so during the hash lookup - by hashing NULLs that occur on the probe side to a different value than NULLs that occur on the build side. The hash table lookup would then (very likely) result in a miss
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Remove nulls in joins during hash table lookup [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #15784:
URL: https://github.com/apache/datafusion/issues/15784#issuecomment-2817826955
Additionally, I think there is an optimization that pushes down null filters down. Maybe it would be worth testing if it can not pushed down to execute it in the join itself (so it doesn't need to loaded in build side or be used for probing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Remove nulls in joins during hash table lookup [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #15784:
URL: https://github.com/apache/datafusion/issues/15784#issuecomment-2817567103
It's a good idea. However, I think if the build side has a large HT and the probe side has many `NULL`s, the existing implementation is likely to be faster. Some statistics might be needed to guide this optimization decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Remove nulls in joins during hash table lookup [datafusion]
Posted by "ctsk (via GitHub)" <gi...@apache.org>.
ctsk commented on issue #15784:
URL: https://github.com/apache/datafusion/issues/15784#issuecomment-2817904944
That does make sense. I found the [filter_null_join_keys](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/filter_null_join_keys.rs) rule which takes care of adding not-null filters to the keys of joins. I thus consider optimizing this case in the hash join a low priority as I expect the improvement to be small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org