You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/15 03:40:03 UTC

[GitHub] [arrow-ballista] r4ntix opened a new issue, #568: Allow `ballista_scheduler` to be embedded in the process of other applications

r4ntix opened a new issue, #568:
URL: https://github.com/apache/arrow-ballista/issues/568

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   Recently, I am working with Ballista as a distributed query engine in our existing scheduling service systems.
   
   I'm trying to embed `ballista_scheduler` into our already existing scheduling system due to the following cases:
   * We have some internal private implementations of `TableProvider` that cannot be serialized into `ballista_scheduler` through the `ballista_client` using protobuf.
   * Need to integrate `ballista_scheduler` and our existing scheduling system into one process.
   
   I found that although `ballista_scheduler` has exposed the `scheduler_server` struct through the `pub mod scheduler_server`, but it still cannot be integrated and embedded. 
   
   Because the `state` and `submit_job` in the `scheduler_server` only are `pub(crate)`:
   https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L59-L66
   https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L150-L167
   
   **Describe the solution you'd like**
   Expose `state` and `submit_job` in the `scheduler_server` to pub.
   
   **Describe alternatives you've considered**
   
   
   **Additional context**
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-ballista] r4ntix commented on issue #568: Allow `ballista_scheduler` to be embedded in the process of other applications

Posted by GitBox <gi...@apache.org>.
r4ntix commented on issue #568:
URL: https://github.com/apache/arrow-ballista/issues/568#issuecomment-1354280942

   @andygrove @yahoNanJing Could you please give some feedback and suggestions?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-ballista] thinkharderdev commented on issue #568: Allow `ballista_scheduler` to be embedded in the process of other applications

Posted by GitBox <gi...@apache.org>.
thinkharderdev commented on issue #568:
URL: https://github.com/apache/arrow-ballista/issues/568#issuecomment-1372460660

   @r4ntix What would you need from `state`? I'm not so sure exposing that as a public interface is a great idea. For the most part the grpc interface has evolved to the point where it is primarily just pushing events into the scheduler's event loop so it wouldn't be major issue to expose those as public methods from `SchedulerServer` itself. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org