You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "robtandy (via GitHub)" <gi...@apache.org> on 2025/01/16 14:56:50 UTC
Re: [I] Fix scalability limitations of current implementation [datafusion-ray]
robtandy commented on issue #46:
URL: https://github.com/apache/datafusion-ray/issues/46#issuecomment-2595948582
I was able to refactor the code such that batches are exchanged in a streaming fashion through the ray object store.
Its too slow though and struggles to scale when run on larger TPCH scale factors. The heavy interaction between all stages and exchanging object refs pointing to batches is too much for Ray. I would observe a typical TPCH query creating 10000+ ray tasks during its execution and the bottle neck appeared to the be the object store.
I'll refactor again trying to exchange data with Arrow Flight as you have suggested @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org