You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "feribg (via GitHub)" <gi...@apache.org> on 2023/10/05 16:00:11 UTC
[I] Expose the lower level scheduler API [arrow-ballista]
feribg opened a new issue, #888:
URL: https://github.com/apache/arrow-ballista/issues/888
I was reading through examples and source but there doesn't seem to be a very obvious way to interact with the scheduler.
Since currently there dont seem to be any good distributed computing libraries in the rust ecosystem and `ray` has no plans to offer official rust support, it would be a very nice addition to be able to interact with the underlying tasks in 2 main ways:
* Submit tasks to the scheduler in the form of runnable rust code (even if for now the DAG is ignored)
* be able to levelrage the data distribution to allow shared data to be handled by the tasks.
maybe there is already a straightforward way to do that in which case im happy to close this and contribute to the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose the lower level scheduler API [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #888:
URL: https://github.com/apache/datafusion-ballista/issues/888#issuecomment-2815650611
closing this as #1243 exposed some way to extend task scheduling, if more is needed happy to discuss and open new issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose the lower level scheduler API [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #888: Expose the lower level scheduler API
URL: https://github.com/apache/datafusion-ballista/issues/888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose the lower level scheduler API [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #888:
URL: https://github.com/apache/datafusion-ballista/issues/888#issuecomment-2814070479
#1243 should expose initial implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org