[GitHub] [arrow-ballista] zhzy0077 commented on issue #714: Submitting same DF to ballista is slow

["zhzy0077 (via GitHub)" <gi...@apache.org>]

zhzy0077 commented on issue #714:
URL: https://github.com/apache/arrow-ballista/issues/714#issuecomment-1486604900

   Interestingly, there are quite a few cases where Ballista pools results from executor and the pool timeout is fixed 100ms. 
   To compare, I ran the same code and see 250ms on out of box code, and see 160ms per loop after applying this patch:
   ```diff
   --- a/ballista/core/src/execution_plans/distributed_query.rs
   +++ b/ballista/core/src/execution_plans/distributed_query.rs
   @@ -259,7 +259,7 @@ async fn execute_query(
                .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?
                .into_inner();
            let status = status.and_then(|s| s.status);
   -        let wait_future = tokio::time::sleep(Duration::from_millis(100));
   +        let wait_future = tokio::time::sleep(Duration::from_millis(1));
            let has_status_change = prev_status != status;
            match status {
                None => {
   ```
   And further to around 70ms per loop with this patch:
   ```diff
   --- a/ballista/executor/src/execution_loop.rs
   +++ b/ballista/executor/src/execution_loop.rs
   @@ -127,7 +127,7 @@ pub async fn poll_loop<T: 'static + AsLogicalPlan, U: 'static + AsExecutionPlan>
            }
    
            if !active_job {
   -            tokio::time::sleep(Duration::from_millis(100)).await;
   +            tokio::time::sleep(Duration::from_millis(1)).await;
            }
        }
    }
   ```
   
   Maybe Ballista should support configuring this wait period?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org