You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sebbegg (via GitHub)" <gi...@apache.org> on 2025/12/18 16:20:24 UTC

[PR] Add arrow flight proxy [datafusion-ballista]

sebbegg opened a new pull request, #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #1349
   
    # Rationale for this change
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   See #1349. Having a proxy on the scheduler makes it easier to e.g. expose this in docker-compose or kubernetes.
   
   Making this a draft first, let me know what you think.
   
   # What changes are included in this PR?
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   This adds a "flight proxy" service to the scheduler that's optionally startet, when `advertise_flight_sql_endpoint` is set.
   This implements only `do_get` and simply proxies the requests to the actual executor.
   I reasoned that having this as a separate service (instead of another method in the scheduler grpc) makes this more flexible since the code on the client side remains almost unchanged - except for the logic to pick scheduler or executor host as the endpoint.
   
   # Are there any user-facing changes?
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   Kind of - using `advertise_flight_sql_endpoint` now actually has an effect.
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2641332985


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {

Review Comment:
   I would ignore pooling for now, we can add it later if you agree



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2678827078


##########
ballista/core/proto/ballista.proto:
##########
@@ -642,6 +642,7 @@ message JobStatus {
 
 message GetJobStatusResult {
   JobStatus status = 1;
+  optional string flight_endpoint = 2;

Review Comment:
   would it make sense to name this as "flight proxy" or similar?  



##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -131,7 +180,24 @@ pub async fn start_server(
 ) -> ballista_core::error::Result<()> {
     info!("Ballista v{BALLISTA_VERSION} Scheduler listening on {address:?}");
     let scheduler =
-        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config).await?;
+        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config.clone())
+            .await?;
 
-    start_grpc_service(address, scheduler).await
+    info!(
+        "advertise_flight_sql_endpoint: {:?}",
+        config.advertise_flight_sql_endpoint
+    );
+    match config.advertise_flight_sql_endpoint {
+        Some(_) => {
+            info!("Starting flight proxy");
+            let flight_proxy = start_flight_proxy_server(config, scheduler.state.clone());

Review Comment:
   Would it make sense to run proxy as a service on the same port with a scheduler service? It would simplify configuration. 



##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -327,7 +328,10 @@ async fn execute_query(
     let mut prev_status: Option<job_status::Status> = None;
 
     loop {
-        let GetJobStatusResult { status } = scheduler
+        let GetJobStatusResult {
+            status,
+            flight_endpoint,

Review Comment:
   would it make sense to support Some("") in which case client should use scheduler address and port ? This way scheduler should not relly care about its public port? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635953109


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   I get that, my worry was about where to fill the information.
   The `SuccessfulJob` and `PartitionLocation` objects are all created in `execution_graph.rs` it seems. It appears weird to somehow forward the proxy information all the way into the execution graph in order to be able to fill a new field like `SuccessfulJob.flight_endpoint`
   
   The alternative could be to clone & update the `SuccessfulJob` in the grpc endpoint:
   
   https://github.com/apache/datafusion-ballista/blob/118327ca424cf14e38e522b7f0af4ed9c6b3c8c3/ballista/scheduler/src/scheduler_server/grpc.rs#L450
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635520998


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   Changing `SuccessfulJob` would require a few changes in more modules… not sure it’s the right place?
   An alternative would be to add it as a second field to `GetJobStatusResult?`
   That would limit the impact to the scheduler grpc server only 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3672638337

   Perhaps @martin-g may be able to help with quick review, I'd be very thankful 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2720367363


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{GrpcClientConfig, create_grpc_client_connection};
+
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+///
+/// The proxy only implements the FlightService::do_get api and forwards the requests
+/// to the respective executors.
+///
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+}
+
+impl BallistaFlightProxyService {
+    pub fn new(
+        max_decoding_message_size: usize,
+        max_encoding_message_size: usize,
+    ) -> Self {
+        Self {
+            max_decoding_message_size,
+            max_encoding_message_size,
+        }
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(
+                    host,
+                    *port,
+                    self.max_decoding_message_size,
+                    self.max_encoding_message_size,
+                )
+                .map_err(|e| from_ballista_err(&e))
+                .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &str,
+    port: u16,
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");

Review Comment:
   its great you catch those, maybe we should fix them with follow up or as part of #1400 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm merged PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3728072457

   thanks @sebbegg will have a look today/tomorrow 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2641337791


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)

Review Comment:
   Same like pooling, do we need this check in the first version?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2634743737


##########
ballista/core/src/serde/generated/ballista.rs:
##########
@@ -1449,6 +1456,36 @@ pub mod scheduler_grpc_client {
                 );
             self.inner.unary(req, path, codec).await
         }
+        /// Provide info on flight proxy address

Review Comment:
   as previous comment, maybe we do not need this endpoint, so we save a request



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635960268


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)
+        .await
+        .map_err(|e| {
+            BallistaError::GrpcConnectionError(format!(
+                "Error connecting to Ballista scheduler or executor at {addr}: {e:?}"
+            ))
+        })?;
+    let flight_client = FlightServiceClient::new(connection)
+        .max_decoding_message_size(16 * 1024 * 1024)

Review Comment:
   Certainly.
   
   No idea on whether this should be a new setting 🤔 
   I'd guess starting with the current one might be fine and if there should be need for a dedicated setting one could revisit?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2639110068


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {
+                    address: flight_proxy_address,
+                } = scheduler
+                    .get_flight_endpoint_info(FlightEndpointInfoParams {})
+                    .await
+                    .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?

Review Comment:
   obsolete with sending the flight-endpoint as part of the job status



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2641108534


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {

Review Comment:
   Looked into that. I mentioned in the issue - I'm new to rust so maybe I'm missing something:
   
   My first attempt for a pool would have been a plain `HashMap<(String, u16), FlightClient>` that just keeps existing clients for executors alive for reuse, with `(host, port)` being the key.
   Unfortunately the `FlightService` trait has all methods as non-mutable. So I can't make `do_get` use `&mut self`...
   
   Is there a smart way around that?
   It seems tonic also doesn't offer pools out of the box...
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2689631076


##########
ballista/core/proto/ballista.proto:
##########
@@ -642,6 +642,7 @@ message JobStatus {
 
 message GetJobStatusResult {
   JobStatus status = 1;
+  optional string flight_proxy = 2;

Review Comment:
   yes something like that, 
   perhaps,
   ```
   oneof flight_proxy {
       bool local = 1; 
       string external = 4;
   }
   ```
   
   something like that 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3750495913

   > Ok, so for the scope of this PR, should we add the extra proxy as an additional executable? A minimalistic approach could be to only implement the embedded variant and leave an external flight-proxy executable up to users.
   
   I agree
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2721790280


##########
ballista/scheduler/src/config.rs:
##########
@@ -41,7 +41,7 @@ pub struct Config {
     /// Route for proxying flight results via scheduler (IP:PORT format).
     #[arg(

Review Comment:
   Thanks 👍 
   Looks good:
   
   ```console
   % cargo run --bin ballista-scheduler -- --advertise-flight-sql-endpoint
       Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
        Running `target/debug/ballista-scheduler --advertise-flight-sql-endpoint`
   2026-01-23T15:44:14.805527Z  INFO main ThreadId(01) ballista_scheduler::scheduler_process: Ballista v51.0.0 Scheduler listening on 0.0.0.0:50050
   2026-01-23T15:44:14.805652Z  INFO main ThreadId(01) ballista_scheduler::scheduler_process: Starting Scheduler grpc server with task scheduling policy of PullStaged
   2026-01-23T15:44:14.808318Z  INFO main ThreadId(01) ballista_scheduler::scheduler_server::query_stage_scheduler: Starting QueryStageScheduler
   2026-01-23T15:44:14.809134Z  INFO tokio-runtime-worker ThreadId(11) ballista_core::event_loop: Starting the event loop query_stage
   2026-01-23T15:44:14.810022Z  INFO                 main ThreadId(01) ballista_scheduler::scheduler_process: Adding embedded flight proxy service on scheduler
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2673441051


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)

Review Comment:
   This appeared to be straight forward, I've added a check that the requested `host:port` belongs to a registered & active executor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3748618528

   > > thanks @sebbegg,
   > > just to clarify, we can have three configuration options:
   > > 
   > > * proxy not configured, client needs to fetc data from executors
   > > * proxy configured, no ip address or port provided, scheduler needs to start proxy on the same port (withing process)
   > > * proxy configured ip/port provided, scheduler considers this as extenal process running proxy, it just needs to put that value in the response, scheduler will not start proxy. client needs to use that ip/port combination to connect to process
   > 
   > If I get this right, the last variant would mean we don't need this block, right?
   > 
   > https://github.com/sebbegg/datafusion-ballista/blob/5022263904c37d660bc77e3f5c065206b6720d20/ballista/scheduler/src/scheduler_process.rs#L202-L212
   > 
   
   yes we don't start in process proxy on a different port
   
   > How would you then start this external process? I guess we could add another crate/binary at `ballista/flight-proxy`?
   
   we can provide new library, or users create their own based on proxy you have created
   
   
   > Starting a cluster could then look like:
   > 
   > * `./ballista-flight-proxy --bind-host localhost --bind-port 50040`
   > * `./ballista-scheduler --advertise-flight-sql-endpoint localhost:50040`
   > * `./ballista-executor --scheduler-host localhost --scheduler-port 50050`
   > 
   > I guess it's smart because like this all services can be run independently.
   
   yes, we offload scheduler process from proxying data, and let it in charge of orchestration only
   
   > As far as I can tell all the scheduler-state is in-memory right? So in this setup we could e.g. not perform the check whether the requested data / executor-host is actually alive and belongs to the cluster. On the other hand, it would make the proxy stateless, which is probably a good thing.
   
   maybe we could relax this requirement, perhaps i should speak earlier. why do we need to check if executor is there? there is no corrective actions we can take. 
   
   > I wonder though, whether it's worthwhile to add the possibility (and hence the complexity in the cli & protobuf) of running the flight-proxy "embedded" in the scheduler?
   
   I'm not sure i understand, we still have option to run it "embedded" 
   
   ```
   * `./ballista-scheduler --advertise-flight-sql-endpoint`
   ```
   
   should listen "embedded".
   please let me know if i got you wrong 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3780922496

   I apologise @sebbegg, I'm catching up with reviews


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635953109


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   I get that, my worry was about where to fill the information.
   The `SuccessfulJob` and `PartitionLocation` objects are all created in `execution_graph.rs` it seems. It appears weird to somehow forward the proxy information all the way into the execution graph in order to be able to fill a new field like `SuccessfulJob.flight_endpoint`
   
   The alternative could be to clone & update the `SuccessfulJob` in the grpc endpoint:
   
   https://github.com/apache/datafusion-ballista/blob/118327ca424cf14e38e522b7f0af4ed9c6b3c8c3/ballista/scheduler/src/scheduler_server/grpc.rs#L450
   
   pseudo:
   ```
   fn get_job_status(job_id) {
     job_status = task_manager.get_job_status(job_id)
     if job_status.status is SuccessfulJob:
         job_status.status.flight_endpoint = self.state.config.advertise_flight_sql_endpoint
     
     return job_status
   }
   ```
   
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2678842263


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -327,7 +328,10 @@ async fn execute_query(
     let mut prev_status: Option<job_status::Status> = None;
 
     loop {
-        let GetJobStatusResult { status } = scheduler
+        let GetJobStatusResult {
+            status,
+            flight_endpoint,

Review Comment:
   not sure should we use Some("") or we have proto enums to represent proxy cases 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2634726171


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   I dont think we should do a round trip to fetch endpoint info. Could we add optional response parameter in `message SuccessfulJob` ? and if it is present do a proxy request? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2641067692


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)

Review Comment:
   Added a check that host+port match with a known (and alive?) executor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2683625333


##########
ballista/core/proto/ballista.proto:
##########
@@ -642,6 +642,7 @@ message JobStatus {
 
 message GetJobStatusResult {
   JobStatus status = 1;
+  optional string flight_proxy = 2;

Review Comment:
   can we make this one of to represent proxy statuses
   
   - no proxy
   - in process (no need to specify ip/port client should use scheduler ip port)
   - external process (ip / port specified) client should use given ip/port
   
   that would remove check if for empty string on client side 



##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::state::SchedulerState;
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{GrpcClientConfig, create_grpc_client_connection};
+use datafusion_proto::logical_plan::AsLogicalPlan;
+use datafusion_proto::physical_plan::AsExecutionPlan;
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::collections::HashSet;
+use std::pin::Pin;
+use std::sync::Arc;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol

Review Comment:
   it would be great if we could add more comments here. describe how it can be configured 



##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -131,7 +191,24 @@ pub async fn start_server(
 ) -> ballista_core::error::Result<()> {
     info!("Ballista v{BALLISTA_VERSION} Scheduler listening on {address:?}");
     let scheduler =
-        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config).await?;
+        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config.clone())
+            .await?;
 
-    start_grpc_service(address, scheduler).await
+    info!(
+        "advertise_flight_sql_endpoint: {:?}",
+        config.advertise_flight_sql_endpoint
+    );
+    match config.advertise_flight_sql_endpoint.clone() {
+        Some(s) if s != "" => {

Review Comment:
   Sorry I might be unclear. If we specify different port (or ip port) that would mean there is external process running proxy, not the scheduler process.
   
   So we have three configuration options
   
   - no proxy 
   - in process (no need to specify ip/port client should use scheduler ip port)
   - external process (ip / port specified) client should use given ip/port



##########
ballista/scheduler/src/scheduler_server/grpc.rs:
##########
@@ -450,7 +450,22 @@ impl<T: 'static + AsLogicalPlan, U: 'static + AsExecutionPlan> SchedulerGrpc
         let job_id = request.into_inner().job_id;
         trace!("Received get_job_status request for job {}", job_id);
         match self.state.task_manager.get_job_status(&job_id).await {
-            Ok(status) => Ok(Response::new(GetJobStatusResult { status })),
+            Ok(status) => Ok(Response::new(GetJobStatusResult {
+                status,
+                flight_proxy: self
+                    .state
+                    .config
+                    .advertise_flight_sql_endpoint
+                    .clone()
+                    .map(|s| match s {
+                        s if s.is_empty() => format!(

Review Comment:
   i guess same thing here, if configuration is empty client should dial back on scheduler address / port 



##########
ballista/scheduler/src/config.rs:
##########
@@ -41,7 +41,7 @@ pub struct Config {
     /// Route for proxying flight results via scheduler (IP:PORT format).
     #[arg(
         long,
-        help = "Route for proxying flight results via scheduler. Should be of the form 'IP:PORT"
+        help = "Route for proxying flight results via scheduler. Should be of the form 'IP:PORT'"

Review Comment:
   same comment regarding empty address. if address/port not specified client need to dial back on scheduler address



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2687841913


##########
ballista/core/proto/ballista.proto:
##########
@@ -642,6 +642,7 @@ message JobStatus {
 
 message GetJobStatusResult {
   JobStatus status = 1;
+  optional string flight_proxy = 2;

Review Comment:
   E.g. like
   ```protobuf
   oneof flight_proxy {
       bool no_proxy = 2;
       bool in_scheduler = 3;
       string external_address = 4;
   }
   ```
   ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3750485670

   > maybe we could relax this requirement, perhaps i should speak earlier. why do we need to check if executor is there? there is no corrective actions we can take.
   
   That was a comment on the PR - but sure, we can drop this.
   
   > we can provide new library, or users create their own based on proxy you have created
   
   Ok, so for the scope of this PR, should we add the extra proxy as an additional executable?
   A minimalistic approach could be to only implement the embedded variant and leave an external flight-proxy executable up to users.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3765807934

   I'll try to review changes tomorrow 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2714406356


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{GrpcClientConfig, create_grpc_client_connection};
+
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+///
+/// The proxy only implements the FlightService::do_get api and forwards the requests
+/// to the respective executors.
+///
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+}
+
+impl BallistaFlightProxyService {
+    pub fn new(
+        max_decoding_message_size: usize,
+        max_encoding_message_size: usize,
+    ) -> Self {
+        Self {
+            max_decoding_message_size,
+            max_encoding_message_size,
+        }
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(
+                    host,
+                    *port,
+                    self.max_decoding_message_size,
+                    self.max_encoding_message_size,
+                )
+                .map_err(|e| from_ballista_err(&e))
+                .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &str,
+    port: u16,
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");

Review Comment:
   we should not assume `http` here 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2714404984


##########
ballista/scheduler/src/config.rs:
##########
@@ -41,7 +41,7 @@ pub struct Config {
     /// Route for proxying flight results via scheduler (IP:PORT format).
     #[arg(

Review Comment:
   ```
   #[arg(
           long,
           num_args = 0..=1,
           default_missing_value = "",
           help = "Route for proxying flight results via scheduler. Use 'HOST:PORT' to let clients fetch results from the specified address. If empty a flight proxy will be started on the scheduler host and port."
       )]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2714219542


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -415,22 +426,75 @@ async fn execute_query(
     }
 }
 
+fn get_client_host_port(
+    executor_metadata: &ExecutorMetadata,
+    scheduler_url: &str,
+    flight_proxy: &Option<FlightProxy>,
+) -> Result<(String, u16)> {
+    fn split_host_port(address: &str) -> Result<(String, u16)> {
+        let url: Url = address.parse().map_err(|e| {
+            DataFusionError::Execution(format!(
+                "Cannot parse host:port in {address:?}: {e}"
+            ))
+        })?;
+        let host = url
+            .host_str()
+            .ok_or(DataFusionError::Execution(format!(
+                "No host in {address:?}"
+            )))?
+            .to_string();
+        let port: u16 = url.port().ok_or(DataFusionError::Execution(format!(
+            "No port in {address:?}"
+        )))?;
+        Ok((host, port))
+    }
+
+    match flight_proxy {
+        Some(FlightProxy::External(address)) => {
+            info!("Fetching results from external flight proxy: {}", address);

Review Comment:
   please use debug



##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -415,22 +426,75 @@ async fn execute_query(
     }
 }
 
+fn get_client_host_port(
+    executor_metadata: &ExecutorMetadata,
+    scheduler_url: &str,
+    flight_proxy: &Option<FlightProxy>,
+) -> Result<(String, u16)> {
+    fn split_host_port(address: &str) -> Result<(String, u16)> {
+        let url: Url = address.parse().map_err(|e| {
+            DataFusionError::Execution(format!(
+                "Cannot parse host:port in {address:?}: {e}"
+            ))
+        })?;
+        let host = url
+            .host_str()
+            .ok_or(DataFusionError::Execution(format!(
+                "No host in {address:?}"
+            )))?
+            .to_string();
+        let port: u16 = url.port().ok_or(DataFusionError::Execution(format!(
+            "No port in {address:?}"
+        )))?;
+        Ok((host, port))
+    }
+
+    match flight_proxy {
+        Some(FlightProxy::External(address)) => {
+            info!("Fetching results from external flight proxy: {}", address);
+            split_host_port(format!("http://{address}").as_str())
+        }
+        Some(FlightProxy::Local(true)) => {
+            info!("Fetching results from scheduler: {}", scheduler_url);

Review Comment:
   please use debug



##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -96,6 +99,24 @@ pub async fn start_grpc_service<
     let mut tonic_builder = RoutesBuilder::default();
     tonic_builder.add_service(scheduler_grpc_server);
 
+    match &config.advertise_flight_sql_endpoint {
+        Some(proxy) if proxy.is_empty() => {
+            info!("Adding embeddded flight proxy service on scheduler");

Review Comment:
   typo `embedded`



##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -415,22 +426,75 @@ async fn execute_query(
     }
 }
 
+fn get_client_host_port(
+    executor_metadata: &ExecutorMetadata,
+    scheduler_url: &str,
+    flight_proxy: &Option<FlightProxy>,
+) -> Result<(String, u16)> {
+    fn split_host_port(address: &str) -> Result<(String, u16)> {
+        let url: Url = address.parse().map_err(|e| {
+            DataFusionError::Execution(format!(
+                "Cannot parse host:port in {address:?}: {e}"
+            ))
+        })?;
+        let host = url
+            .host_str()
+            .ok_or(DataFusionError::Execution(format!(
+                "No host in {address:?}"
+            )))?
+            .to_string();
+        let port: u16 = url.port().ok_or(DataFusionError::Execution(format!(
+            "No port in {address:?}"
+        )))?;
+        Ok((host, port))
+    }
+
+    match flight_proxy {
+        Some(FlightProxy::External(address)) => {
+            info!("Fetching results from external flight proxy: {}", address);
+            split_host_port(format!("http://{address}").as_str())
+        }
+        Some(FlightProxy::Local(true)) => {
+            info!("Fetching results from scheduler: {}", scheduler_url);
+            split_host_port(scheduler_url)
+        }
+        Some(FlightProxy::Local(false)) | None => {
+            info!(

Review Comment:
   please use debug



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2719845556


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{GrpcClientConfig, create_grpc_client_connection};
+
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+///
+/// The proxy only implements the FlightService::do_get api and forwards the requests
+/// to the respective executors.
+///
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+}
+
+impl BallistaFlightProxyService {
+    pub fn new(
+        max_decoding_message_size: usize,
+        max_encoding_message_size: usize,
+    ) -> Self {
+        Self {
+            max_decoding_message_size,
+            max_encoding_message_size,
+        }
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(
+                    host,
+                    *port,
+                    self.max_decoding_message_size,
+                    self.max_encoding_message_size,
+                )
+                .map_err(|e| from_ballista_err(&e))
+                .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &str,
+    port: u16,
+    max_decoding_message_size: usize,
+    max_encoding_message_size: usize,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");

Review Comment:
   Hm, other usages of `create_grpc_client_connection` follow the same usage:
   
   https://github.com/apache/datafusion-ballista/blob/caea92969f34a0c0899a6eb13713b9f7e377f80d/ballista/core/src/client.rs#L73-L76
   
   https://github.com/apache/datafusion-ballista/blob/caea92969f34a0c0899a6eb13713b9f7e377f80d/ballista/executor/src/executor_process.rs#L201
   
   It's somewhat inconsistent that some parts of the code use host+port while at other places require `Url`s or url-like strings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2684919973


##########
ballista/scheduler/src/scheduler_server/grpc.rs:
##########
@@ -450,7 +450,22 @@ impl<T: 'static + AsLogicalPlan, U: 'static + AsExecutionPlan> SchedulerGrpc
         let job_id = request.into_inner().job_id;
         trace!("Received get_job_status request for job {}", job_id);
         match self.state.task_manager.get_job_status(&job_id).await {
-            Ok(status) => Ok(Response::new(GetJobStatusResult { status })),
+            Ok(status) => Ok(Response::new(GetJobStatusResult {
+                status,
+                flight_proxy: self
+                    .state
+                    .config
+                    .advertise_flight_sql_endpoint
+                    .clone()
+                    .map(|s| match s {
+                        s if s.is_empty() => format!(

Review Comment:
   That's what happens - just felt that the "switch" was easier to implement on scheduler side.
   This way there's a bit less logic on the client side. Can move this though.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2685004927


##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -131,7 +191,24 @@ pub async fn start_server(
 ) -> ballista_core::error::Result<()> {
     info!("Ballista v{BALLISTA_VERSION} Scheduler listening on {address:?}");
     let scheduler =
-        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config).await?;
+        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config.clone())
+            .await?;
 
-    start_grpc_service(address, scheduler).await
+    info!(
+        "advertise_flight_sql_endpoint: {:?}",
+        config.advertise_flight_sql_endpoint
+    );
+    match config.advertise_flight_sql_endpoint.clone() {
+        Some(s) if s != "" => {

Review Comment:
   Yes, but you need to specify advertising address and configure it correctly, which may be tricky in docker containers. 
   
   Suggestion would eliminate that as client already knows scheduler address. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3746283087

   > thanks @sebbegg,
   > 
   > just to clarify, we can have three configuration options:
   > 
   > * proxy not configured, client needs to fetc data from executors
   > * proxy configured, no ip address or port provided, scheduler needs to start proxy on the same port (withing process)
   > * proxy configured ip/port provided, scheduler considers this as extenal process running proxy, it just needs to put that value in the response, scheduler will not start proxy. client needs to use that ip/port combination to connect to process
   
   If I get this right, the last variant would mean we don't need this block, right?
   
   https://github.com/sebbegg/datafusion-ballista/blob/5022263904c37d660bc77e3f5c065206b6720d20/ballista/scheduler/src/scheduler_process.rs#L202-L212
   
   How would you then start this external process?
   I guess we could add another crate/binary at `ballista/flight-proxy`?
   
   Starting a cluster could then look like:
   
   * `./ballista-flight-proxy --bind-host localhost --bind-port 50040`
   * `./ballista-scheduler --advertise-flight-sql-endpoint localhost:50040`
   * `./ballista-executor --scheduler-host localhost --scheduler-port 50050`
   
   I guess it's smart because like this all services can be run independently.
   
   As far as I can tell all the scheduler-state is in-memory right?
   So in this setup we could e.g. not perform the check whether the requested data / executor-host is actually alive and belongs to the cluster.
   On the other hand, it would make the proxy stateless, which is probably a good thing.
   
   I wonder though, whether it's worthwhile to add the possibility (and hence the complexity in the cli & protobuf) of running the flight-proxy "embedded" in the scheduler?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635953109


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   I get that, my worry was about where to fill the information.
   The `SuccessfulJob` and `PartitionLocation` objects are all created in `execution_graph.rs` it seems. It appears weird to somehow forward the proxy information all the way into the execution graph.
   
   The alternative could be to clone & update the `SuccessfulJob` in the grpc endpoint:
   
   https://github.com/apache/datafusion-ballista/blob/118327ca424cf14e38e522b7f0af4ed9c6b3c8c3/ballista/scheduler/src/scheduler_server/grpc.rs#L450
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3714123886

   hey @sebbegg is there anything i can do to help you with this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3714152505

   Didn’t find the time I hoped for during the holidays…I’ll try to get back to it this week.Am 06.01.2026 um 11:23 schrieb Marko Milenković ***@***.***>:milenkovicm left a comment (apache/datafusion-ballista#1351)
   hey @sebbegg is there anything i can do to help you with this PR?
   
   —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2634001951


##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,

Review Comment:
   ```suggestion
       host: &str,
   ```



##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {
+                    address: flight_proxy_address,
+                } = scheduler
+                    .get_flight_endpoint_info(FlightEndpointInfoParams {})
+                    .await
+                    .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?

Review Comment:
   Thinking out loud: Does it need to fail here or it is OK to log it and fallback to no-proxy ?



##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -131,7 +176,19 @@ pub async fn start_server(
 ) -> ballista_core::error::Result<()> {
     info!("Ballista v{BALLISTA_VERSION} Scheduler listening on {address:?}");
     let scheduler =
-        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config).await?;
+        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config.clone())
+            .await?;
 
-    start_grpc_service(address, scheduler).await
+    info!(
+        "advertise_flight_sql_endpoint: {:?}",
+        config.advertise_flight_sql_endpoint
+    );
+    match config.advertise_flight_sql_endpoint {
+        Some(_) => {
+            info!("Starting flight proxy");
+            let _flight_proxy = start_flight_proxy_server(config);

Review Comment:
   This ignores any errors returned by the the proxy. They are logged, so this might be intentional - to continue working even without the proxy.
   
   If both services should fail if either of them fail then you could use `select!()`. E.g.:
   ```diff
   -            let _flight_proxy = start_flight_proxy_server(config);
   -            start_grpc_service(address, scheduler).await
   +            let flight_proxy = start_flight_proxy_server(config);
   +            tokio::select! {
   +                result = start_grpc_service(address, scheduler) => result,
   +                result = flight_proxy => {
   +                    result.map_err(|e| BallistaError::Internal(format!("Flight proxy task panicked: {e:?}")))?
   +                }
   +            }
   ```
   
   Also there is a racing here - the main service may start faster and any requests won't be routed thru the proxy until it fully starts too.



##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)
+        .await
+        .map_err(|e| {
+            BallistaError::GrpcConnectionError(format!(
+                "Error connecting to Ballista scheduler or executor at {addr}: {e:?}"
+            ))
+        })?;
+    let flight_client = FlightServiceClient::new(connection)
+        .max_decoding_message_size(16 * 1024 * 1024)

Review Comment:
   Can/should we use `config.grpc_server_max_encoding_message_size` or a new setting ?
   Same for `min` below.



##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,

Review Comment:
   ```suggestion
       port: u16,
   ```



##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -121,6 +123,49 @@ pub async fn start_grpc_service<
         .map_err(BallistaError::from)
 }
 
+fn start_flight_proxy_server(
+    config: Arc<SchedulerConfig>,
+) -> JoinHandle<Result<(), BallistaError>> {
+    tokio::spawn(async move {
+        let address = match config.advertise_flight_sql_endpoint.clone() {
+            Some(flight_sql_endpoint) => flight_sql_endpoint
+                .parse::<SocketAddr>()
+                .map_err(|e: std::net::AddrParseError| {
+                    error!(
+                        "Error parsing advertise_flight_sql_endpoint: {}",
+                        e.to_string()
+                    );
+                    BallistaError::Configuration(e.to_string())
+                })?,
+            _ => {
+                return Err(BallistaError::Configuration(
+                    "Expected advertise flight sql endpoint".into(),
+                ));
+            }
+        };
+
+        let max_encoding_message_size =
+            config.grpc_server_max_encoding_message_size as usize;
+        let max_decoding_message_size =
+            config.grpc_server_max_decoding_message_size as usize;
+        info!("Built-in arrow flight server proxy listening on: {address:?} max_encoding_size: {max_encoding_message_size} max_decoding_size: {max_decoding_message_size}");
+
+        let grpc_server_config = GrpcServerConfig::default();
+        let server_future = create_grpc_server(&grpc_server_config)

Review Comment:
   There is no authentication layer.
   But there is no authentication for the main service too, so this is not required at the moment.



##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {
+    let addr = format!("http://{host}:{port}");
+    let grpc_config = GrpcClientConfig::default();
+    debug!("FlightProxyService connecting to {addr}");
+    let connection = create_grpc_client_connection(addr.clone(), &grpc_config)

Review Comment:
   Since the host & port come from client provided ticket I think it would be good to check somehow that they point to a registered executor. Otherwise it might be used for Server-Side Request Forgery attack.



##########
ballista/scheduler/src/flight_proxy_service.rs:
##########
@@ -0,0 +1,157 @@
+use arrow_flight::flight_service_client::FlightServiceClient;
+use arrow_flight::flight_service_server::FlightService;
+use arrow_flight::{
+    Action, ActionType, Criteria, Empty, FlightData, FlightDescriptor, FlightInfo,
+    HandshakeRequest, HandshakeResponse, PollInfo, PutResult, SchemaResult, Ticket,
+};
+use ballista_core::error::BallistaError;
+use ballista_core::serde::decode_protobuf;
+use ballista_core::serde::scheduler::Action as BallistaAction;
+use ballista_core::utils::{create_grpc_client_connection, GrpcClientConfig};
+use futures::{Stream, TryFutureExt};
+use log::debug;
+use std::pin::Pin;
+use tonic::{Request, Response, Status, Streaming};
+
+/// Service implementing a proxy from scheduler to executor Apache Arrow Flight Protocol
+#[derive(Clone)]
+pub struct BallistaFlightProxyService {}
+
+impl BallistaFlightProxyService {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for BallistaFlightProxyService {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+type BoxedFlightStream<T> =
+    Pin<Box<dyn Stream<Item = Result<T, Status>> + Send + 'static>>;
+
+#[tonic::async_trait]
+impl FlightService for BallistaFlightProxyService {
+    type DoActionStream = BoxedFlightStream<arrow_flight::Result>;
+    type DoExchangeStream = BoxedFlightStream<FlightData>;
+    type DoGetStream = BoxedFlightStream<FlightData>;
+    type DoPutStream = BoxedFlightStream<PutResult>;
+    type HandshakeStream = BoxedFlightStream<HandshakeResponse>;
+    type ListActionsStream = BoxedFlightStream<ActionType>;
+    type ListFlightsStream = BoxedFlightStream<FlightInfo>;
+    async fn handshake(
+        &self,
+        _request: Request<Streaming<HandshakeRequest>>,
+    ) -> Result<Response<Self::HandshakeStream>, Status> {
+        Err(Status::unimplemented("handshake"))
+    }
+
+    async fn list_flights(
+        &self,
+        _request: Request<Criteria>,
+    ) -> Result<Response<Self::ListFlightsStream>, Status> {
+        Err(Status::unimplemented("list_flights"))
+    }
+
+    async fn get_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<FlightInfo>, Status> {
+        Err(Status::unimplemented("get_flight_info"))
+    }
+
+    async fn poll_flight_info(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<PollInfo>, Status> {
+        Err(Status::unimplemented("poll_flight_info"))
+    }
+
+    async fn get_schema(
+        &self,
+        _request: Request<FlightDescriptor>,
+    ) -> Result<Response<SchemaResult>, Status> {
+        Err(Status::unimplemented("get_schema"))
+    }
+
+    async fn do_get(
+        &self,
+        request: Request<Ticket>,
+    ) -> Result<Response<Self::DoGetStream>, Status> {
+        let ticket = request.into_inner();
+
+        let action =
+            decode_protobuf(&ticket.ticket).map_err(|e| from_ballista_err(&e))?;
+
+        match &action {
+            BallistaAction::FetchPartition {
+                host, port, job_id, ..
+            } => {
+                debug!("Fetching results for job id: {job_id} from {host}:{port}");
+                let mut client = get_flight_client(host, port)
+                    .map_err(|e| from_ballista_err(&e))
+                    .await?;
+                client
+                    .do_get(Request::new(ticket))
+                    .await
+                    .map(|r| Response::new(Box::pin(r.into_inner()) as Self::DoGetStream))
+            }
+        }
+    }
+
+    async fn do_put(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoPutStream>, Status> {
+        Err(Status::unimplemented("do_put"))
+    }
+
+    async fn do_exchange(
+        &self,
+        _request: Request<Streaming<FlightData>>,
+    ) -> Result<Response<Self::DoExchangeStream>, Status> {
+        Err(Status::unimplemented("do_exchange"))
+    }
+
+    async fn do_action(
+        &self,
+        _request: Request<Action>,
+    ) -> Result<Response<Self::DoActionStream>, Status> {
+        Err(Status::unimplemented("do_action"))
+    }
+
+    async fn list_actions(
+        &self,
+        _request: Request<Empty>,
+    ) -> Result<Response<Self::ListActionsStream>, Status> {
+        Err(Status::unimplemented("list_actions"))
+    }
+}
+
+fn from_ballista_err(e: &ballista_core::error::BallistaError) -> Status {
+    Status::internal(format!("Ballista Error: {e:?}"))
+}
+
+async fn get_flight_client(
+    host: &String,
+    port: &u16,
+) -> Result<FlightServiceClient<tonic::transport::channel::Channel>, BallistaError> {

Review Comment:
   Consider connection pooling. Currently a new client is created for each request.



##########
ballista/core/proto/ballista.proto:
##########
@@ -707,6 +707,12 @@ message RunningTaskInfo {
   uint32 partition_id = 4;;

Review Comment:
   ```suggestion
     uint32 partition_id = 4;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635953109


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   I get that, my worry was about where to fill the information.
   The `SuccessfulJob` and `PartitionLocation` objects are all created in `execution_graph.rs` it seems. It appears weird to somehow forward the proxy information all the way into the execution graph in order to be able to fill a new field like `SuccessfulJob.flight_endpoint`
   
   The alternative could be to clone & update the `SuccessfulJob` in the grpc endpoint:
   
   https://github.com/apache/datafusion-ballista/blob/118327ca424cf14e38e522b7f0af4ed9c6b3c8c3/ballista/scheduler/src/scheduler_server/grpc.rs#L450
   
   pseudo:
   ```
   job_status = task_manager.get_job_status(job_id)
   if job_status.status is SuccessfulJob:
       job_status.status.flight_endpoint = self.state.config.advertise_flight_sql_endpoint
   
   return job_status
   ```
   
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2635570471


##########
ballista/core/src/execution_plans/distributed_query.rs:
##########
@@ -360,9 +363,22 @@ async fn execute_query(
                 let duration = Duration::from_millis(duration);
 
                 info!("Job {job_id} finished executing in {duration:?} ");
+                let FlightEndpointInfo {

Review Comment:
   https://github.com/apache/datafusion-ballista/blob/118327ca424cf14e38e522b7f0af4ed9c6b3c8c3/ballista/core/src/execution_plans/distributed_query.rs#L356
   
   gives you back partition locations which indicates where data is, so adding optional proxy parameter you could ask it "give me data from `partition_location`" otherwise you just fetch it as it is at the moment (if proxy not provided)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add arrow flight proxy [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3672634276

   Thanks @sebbegg, I'm a bit stuck time wise will try to have a look tomorrow if not will follow up over holiday period


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#issuecomment-3725540036

   @milenkovicm Feel free to have another look - made some updates:
   
   * The `GetJobStatusResult` now has the `flight_endpoint` - so there's no extra request involved to fetch this information.
   * As @martin-g suggested the proxy now checks that the requested `host:port` belongs to an active executor
   * Used `tokio::select!` to exit in case the flight proxy panics
   
   Unfortunately this is still missing tests.
   A proper test would probably be some sort of integration test with scheduler and at least one executor.
   I looked through the test-utils, but I'm not sure there's something that could be used for that...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add arrow flight proxy to scheduler [datafusion-ballista]

Posted by "sebbegg (via GitHub)" <gi...@apache.org>.
sebbegg commented on code in PR #1351:
URL: https://github.com/apache/datafusion-ballista/pull/1351#discussion_r2684965797


##########
ballista/scheduler/src/scheduler_process.rs:
##########
@@ -131,7 +191,24 @@ pub async fn start_server(
 ) -> ballista_core::error::Result<()> {
     info!("Ballista v{BALLISTA_VERSION} Scheduler listening on {address:?}");
     let scheduler =
-        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config).await?;
+        create_scheduler::<LogicalPlanNode, PhysicalPlanNode>(cluster, config.clone())
+            .await?;
 
-    start_grpc_service(address, scheduler).await
+    info!(
+        "advertise_flight_sql_endpoint: {:?}",
+        config.advertise_flight_sql_endpoint
+    );
+    match config.advertise_flight_sql_endpoint.clone() {
+        Some(s) if s != "" => {

Review Comment:
   That’s what happens though?
   This just puts the logic to use the scheduler host:port in the scheduler rather than the client.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org