[jira] [Updated] (KUDU-3364) ThreadPool Timer to execute some Periodic tasks and multi control send tasks

["shenxingwuying (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

shenxingwuying updated KUDU-3364:
---------------------------------
    Description: 
h1. Scenanios

In general, I am talking about a category of problem.

There are some periodic tasks or automatically triggered scheduling tasks in kudu. 

For example, automatic rebalance of cluster data, some GC task and compaction tasks.

Their implementation is by kudu Thread, maybe std::thread or ThreadPool, the really task internally periodic scheduled or internally strategy to trigge execution. 

They are all internal, we cann't do some.

In fact, we need a method our control to trigge the above types of actions.

In general, I am talking about a category of problem. 
Some scenarios is significant.

Below is examples:

 
h2. data rebalance

There are two rebalance ways:

1. enable auto rebalance
2. use rebalance tool 1.14 before.

The two ways maybe exist some conflicts at opeations race, because rebalance tool' logic is a litte complex at tool and auto rebalance is running at master.

In future, auto rebalance at master will become very steady and become the main way for data rebalance. And at the same time, admin opers need a external trigger the rebalance just like auto rebalance.

But, now auto rebalance is running in a thread and by time period.
Although we can add a api for MasterService, but the api is synchronize, and will cose very much, we need a asynchronized method to trigger the rebalance.
h2. auto compaction

Another example is auto compaction,
I have found compaction strategy is not always valid, so maybe we need a method  controlled by admin users to triggle compaction.

If we can do a RowSetInCompaction, we need not restart the kudu cluster.
h1.  
h1. My Solution

Add a timer in ThreadPool. This timer is a worker thread that schedules tasks to the specified thread according to time.

We can limit only SERIAL ThreadPoolToken can enable TimerThread.
Pseudo code expresses my intention：
{code:java}
//代码占位符
class TimerThread {
class Task {         
ThreadPoolToken token;         
std::function<void()> f;     
};
    
void Schedule(Task task, int delay_ms) {         
  tasks_.insert(...);     
}
void RunLoop() {
  while (...) {
    SleepFor(100ms);
    tasks = FindTasks();
    for (auto task : tasks) {
      token = task.token;
      token->Submit(task.f);
      tasks_.erase...             
    }
  }
}
  scoped_refptr<Thread> thread_;
  std::multimap<MonoTime, Task>  tasks;
};

class ThreadPool{  
...  
TimerThread* timer_;
... 
};

class ThreadPoolToken {
  void Scheduler();      
};{code}
This scheme can be compatible with the previous ThreadPool, and timer is nullptr by default.

For periodic tasks, We can use a Control ThreadPool with timer to refact some codes to make them more clear, to avoid the problem of too many single threads in the past.

  was:
# Scenanios
In general, I am talking about a category of problem.

There are some periodic tasks or automatically triggered scheduling tasks in kudu. 

For example, automatic rebalance of cluster data, some GC task and compaction tasks.

Their implementation is by kudu Thread, maybe std::thread or ThreadPool, the really task internally periodic scheduled or internally strategy to trigge execution. 

They are all internal, we cann't do some.

In fact, we need a method our control to trigge the above types of actions.

In general, I am talking about a category of problem. 
Some scenarios is significant.

Below is examples:
## data rebalance
There are two rebalance ways:

1. enable auto rebalance
2. use rebalance tool 1.14 before.

The two ways maybe exist some conflicts at opeations race, because rebalance tool' logic is a litte complex at tool and auto rebalance is running at master.

In future, auto rebalance at master will become very steady and become the main way for data rebalance. And at the same time, admin opers need a external trigger the rebalance just like auto rebalance.

But, now auto rebalance is running in a thread and by time period.
Although we can add a api for MasterService, but the api is synchronize, and will cose very much, we need a asynchronized method to trigger the rebalance.

## auto compaction

Another example is auto compaction,
I have found compaction strategy is not always valid, so maybe we need a method  controlled by admin users to triggle compaction.

If we can do a RowSetInCompaction, we need not restart the kudu cluster.


# My Solution
Add a timer in ThreadPool. This timer is a worker thread that schedules tasks to the specified thread according to time.

We can limit only SERIAL ThreadPoolToken can enable TimerThread.
Pseudo code expresses my intention：
```
class TimerThread {
    class Task {
        ThreadPoolToken token;
        std::function<void()> f;
    }
    
    void Schedule(Task task, int delay_ms) {
        tasks_.insert(...);
    }
    void RunLoop() {
        while (...) {
            SleepFor(100ms);
            
            tasks = FindTasks();
            for (auto task : tasks) {
                token = task.token;
                token->Submit(task.f);
                tasks_.erase...
            }
        }
    }

  scoped_refptr<Thread> thread_;
  std::multimap<MonoTime, Task>  tasks;
};

class ThreadPool {
 ...
 TimerThread* timer_;   
 ...
}

class ThreadPoolToken {
    
    void Scheduler();
    
}

```

This scheme can be compatible with the previous ThreadPool, and timer is nullptr by default.

For periodic tasks, We can use a Control ThreadPool with timer to refact some codes to make them more clear, to avoid the problem of too many single threads in the past.


> ThreadPool Timer to execute some Periodic tasks and multi control send tasks
> ----------------------------------------------------------------------------
>
>                 Key: KUDU-3364
>                 URL: https://issues.apache.org/jira/browse/KUDU-3364
>             Project: Kudu
>          Issue Type: New Feature
>            Reporter: shenxingwuying
>            Assignee: shenxingwuying
>            Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> h1. Scenanios
> In general, I am talking about a category of problem.
> There are some periodic tasks or automatically triggered scheduling tasks in kudu. 
> For example, automatic rebalance of cluster data, some GC task and compaction tasks.
> Their implementation is by kudu Thread, maybe std::thread or ThreadPool, the really task internally periodic scheduled or internally strategy to trigge execution. 
> They are all internal, we cann't do some.
> In fact, we need a method our control to trigge the above types of actions.
> In general, I am talking about a category of problem. 
> Some scenarios is significant.
> Below is examples:
>  
> h2. data rebalance
> There are two rebalance ways:
> 1. enable auto rebalance
> 2. use rebalance tool 1.14 before.
> The two ways maybe exist some conflicts at opeations race, because rebalance tool' logic is a litte complex at tool and auto rebalance is running at master.
> In future, auto rebalance at master will become very steady and become the main way for data rebalance. And at the same time, admin opers need a external trigger the rebalance just like auto rebalance.
> But, now auto rebalance is running in a thread and by time period.
> Although we can add a api for MasterService, but the api is synchronize, and will cose very much, we need a asynchronized method to trigger the rebalance.
> h2. auto compaction
> Another example is auto compaction,
> I have found compaction strategy is not always valid, so maybe we need a method  controlled by admin users to triggle compaction.
> If we can do a RowSetInCompaction, we need not restart the kudu cluster.
> h1.  
> h1. My Solution
> Add a timer in ThreadPool. This timer is a worker thread that schedules tasks to the specified thread according to time.
> We can limit only SERIAL ThreadPoolToken can enable TimerThread.
> Pseudo code expresses my intention：
> {code:java}
> //代码占位符
> class TimerThread {
> class Task {         
> ThreadPoolToken token;         
> std::function<void()> f;     
> };
>     
> void Schedule(Task task, int delay_ms) {         
>   tasks_.insert(...);     
> }
> void RunLoop() {
>   while (...) {
>     SleepFor(100ms);
>     tasks = FindTasks();
>     for (auto task : tasks) {
>       token = task.token;
>       token->Submit(task.f);
>       tasks_.erase...             
>     }
>   }
> }
>   scoped_refptr<Thread> thread_;
>   std::multimap<MonoTime, Task>  tasks;
> };
> class ThreadPool{  
> ...  
> TimerThread* timer_;
> ... 
> };
> class ThreadPoolToken {
>   void Scheduler();      
> };{code}
> This scheme can be compatible with the previous ThreadPool, and timer is nullptr by default.
> For periodic tasks, We can use a Control ThreadPool with timer to refact some codes to make them more clear, to avoid the problem of too many single threads in the past.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)