You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "robtandy (via GitHub)" <gi...@apache.org> on 2025/02/18 15:57:22 UTC
[PR] Preallocate Ray Workers [datafusion-ray]
robtandy opened a new pull request, #62:
URL: https://github.com/apache/datafusion-ray/pull/62
Significant time is spent allocating `StageService`s as the python actor, `RayStage` and waiting for them to bind to a listening port.
This changes the semantics of `RayContext` and `RayDataFrame` such that the `RayQuerySupervisor` is now created when the `RayContext` is created and a pool of `RayStage`s are preallocated. When a `RayDataFrame` is created by the context's `sql()` method, stages are calculated and the number of `RayStage` actors are requested from the pool. When the query is finished, instead of tearing down these actors, they are simply returned to the pool.
The pool size is parameterized by min size and max size values. The pool will preallocate at the minimum size and can grow up to the maximum size. Requesting workers beyond the maximum size will raise an exception. The pool is released and ray resources are torn down when the `RayContext` goes out of scope.
This change makes a significant difference on TPCH benchmarks. Tested on SF100, it improved the result by 25% on a machine with very fast disk, such that the overhead of creating and tearing down ray resources was a large chunk of execution time.
This PR does not handle the pool shrinking back to a minimum size only growing, let's handle that in a subsequent change.
The `tpcbench.py` benchmark script, and `tpc.py` script accept `--worker-pool-min`
As `RayStage` actors are now longer lived, they were updated to be able to accept updated `ExecutionPlan`s to serve. This meant that debugging issues with `RayStage`s is a little more difficult as it no longer makes sense to name them after they stage they are hosting, because that can change. As such, they now receive friendly human readable unique names which make reading debug and info output much easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Preallocate Ray Workers [datafusion-ray]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #62:
URL: https://github.com/apache/datafusion-ray/pull/62#discussion_r1960168520
##########
requirements-in.txt:
##########
@@ -7,6 +7,6 @@ numpy
pyarrow>=18.0.0
pytest
ray==2.40.0
-datafusion==43.1.0
+datafusion==43.0.0
Review Comment:
There is no 43.0.0 release in PyPi, so I think we need to leave this as 43.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Preallocate Ray Workers [datafusion-ray]
Posted by "robtandy (via GitHub)" <gi...@apache.org>.
robtandy commented on PR #62:
URL: https://github.com/apache/datafusion-ray/pull/62#issuecomment-2666307285
Updated to squash messy commit history from source branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Preallocate Ray Workers [datafusion-ray]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #62:
URL: https://github.com/apache/datafusion-ray/pull/62
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Preallocate Ray Workers [datafusion-ray]
Posted by "robtandy (via GitHub)" <gi...@apache.org>.
robtandy commented on code in PR #62:
URL: https://github.com/apache/datafusion-ray/pull/62#discussion_r1960252067
##########
requirements-in.txt:
##########
@@ -7,6 +7,6 @@ numpy
pyarrow>=18.0.0
pytest
ray==2.40.0
-datafusion==43.1.0
+datafusion==43.0.0
Review Comment:
Good eye. Corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org