[GitHub] [arrow-ballista] andygrove commented on a diff in pull request #188: Clean up job data on both Scheduler and Executor

[GitBox <gi...@apache.org>]

andygrove commented on code in PR #188:
URL: https://github.com/apache/arrow-ballista/pull/188#discussion_r985255568


##########
ballista/rust/scheduler/src/state/task_manager.rs:
##########
@@ -43,17 +43,17 @@ use rand::{thread_rng, Rng};
 use std::collections::HashMap;
 use std::default::Default;
 use std::sync::Arc;
+use std::time::Duration;
 use tokio::sync::RwLock;
-use tonic::transport::Channel;
 
-type ExecutorClients = Arc<RwLock<HashMap<String, ExecutorGrpcClient<Channel>>>>;
 type ExecutionGraphCache = Arc<RwLock<HashMap<String, Arc<RwLock<ExecutionGraph>>>>>;
 
+const CLEANUP_FINISHED_JOB_DELAY_SECS: u64 = 300;

Review Comment:
   We should make this configurable. Some of the queries I am testing take much longer than 300 seconds. We already have the ability to set configs on the context.



##########
ballista/rust/scheduler/src/state/task_manager.rs:
##########
@@ -43,17 +43,17 @@ use rand::{thread_rng, Rng};
 use std::collections::HashMap;
 use std::default::Default;
 use std::sync::Arc;
+use std::time::Duration;
 use tokio::sync::RwLock;
-use tonic::transport::Channel;
 
-type ExecutorClients = Arc<RwLock<HashMap<String, ExecutorGrpcClient<Channel>>>>;
 type ExecutionGraphCache = Arc<RwLock<HashMap<String, Arc<RwLock<ExecutionGraph>>>>>;
 
+const CLEANUP_FINISHED_JOB_DELAY_SECS: u64 = 300;

Review Comment:
   Never mind, this is a delay _after_ the job completes. I would still like to see this configurable but we could do that as a follow in PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org