[GitHub] [arrow-datafusion] jon-chuang commented on issue #1221: Task assignment between Scheduler and Executors

[GitBox <gi...@apache.org>]

jon-chuang commented on issue #1221:
URL: https://github.com/apache/arrow-datafusion/issues/1221#issuecomment-1012935879


   Hi all, I've been working on a Rust API for the Ray distributed computing framework that powers many popular python ML libraries like RLLib, Ray Train and Ray Tune.
   
   It is currently nearing the end of the prototype phase and we are looking for real-world usage for the project. You can view the progress here: https://github.com/ray-project/ray/pull/21572
   
   I'm quite interested in exploring the use of the Ray for highly-performant and efficient scheduling of tasks for Ballista. Note that one can do data-aware scheduling with Ray that can be efficient even without a randomized data partitioning - which could open up new performance possibilities. 
   
   Do let me know if anyone is interested in this. I will be happy to chat.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org