[GitHub] [hadoop-ozone] GlenGeng commented on pull request #1518: HDDS-4386: Each EndpointStateMachine uses its own thread pool to talk with SCM/Recon

[GitBox <gi...@apache.org>]

GlenGeng commented on pull request #1518:
URL: https://github.com/apache/hadoop-ozone/pull/1518#issuecomment-715656741


   @avijayanhwx Thanks for looking at this PR. 
   @ChenSammi Could you please create a Jira for slow Recon ?
   
   > Also, irrespective of whether there is a slow SCM or Recon, the DN should not submit a request to an endpoint if there is a pending request to that endpoint. Is that not true currently?
   
   It is not true.
   
   `DatanodeStateMachine.stateMachineThread` calls `context.execute(executorService, heartbeatFrequency, TimeUnit.MILLISECONDS);` every heartbeatFrequency. 
   
   The job is
   ```
   DatanodeState<DatanodeStateMachine.DatanodeStates> task = getTask();
   task.execute(service);
   DatanodeStateMachine.DatanodeStates newState = task.await(time, unit);
   ```
   where the task is either `InitDatanodeState` or `RunningDatanodeState`, depending on state of Datanode state machine:
   ```
     public DatanodeState<DatanodeStateMachine.DatanodeStates> getTask() {
       switch (this.state) {
       case INIT:
         return new InitDatanodeState(this.conf, parent.getConnectionManager(),
             this);
       case RUNNING:
         return new RunningDatanodeState(this.conf, parent.getConnectionManager(),
             this);
       case SHUTDOWN:
         return null;
       default:
         throw new IllegalArgumentException("Not Implemented yet.");
       }
     }
   ```
   
   Say it is `RunningDatanodeState`, it will do a endpointTask for each endpoint, which does not consider whether there is an outstanding request for that endpoint.
   ```
     public void execute(ExecutorService executor) {
       ecs = new ExecutorCompletionService<>(executor);
       for (EndpointStateMachine endpoint : connectionManager.getValues()) {
         Callable<EndPointStates> endpointTask = getEndPointTask(endpoint);
         if (endpointTask != null) {
           ecs.submit(endpointTask);
         } else {
           // This can happen if a task is taking more time than the timeOut
           // specified for the task in await, and when it is completed the task
           // has set the state to Shutdown, we may see the state as shutdown
           // here. So, we need to Shutdown DatanodeStateMachine.
           LOG.error("State is Shutdown in RunningDatanodeState");
           context.setState(DatanodeStateMachine.DatanodeStates.SHUTDOWN);
         }
       }
     }
   ```
   
   and the 'await()' will do a timely wait, so that it will not block the next heartbeat task of DatanodeStateMachine
   ```
     public DatanodeStateMachine.DatanodeStates
         await(long duration, TimeUnit timeUnit)
         throws InterruptedException {
   
       ....
       while (returned < count && timeLeft > 0) {
         Future<EndPointStates> result =
             ecs.poll(timeLeft, TimeUnit.MILLISECONDS);
         if (result != null) {
           results.add(result);
           returned++;
         }
         timeLeft = durationMS - (Time.monotonicNow() - startTime);
       }
       return computeNextContainerState(results);
     }
   ```
   
   Consider `HeartbeatEndpointTask`, its `call()` just acquire the lock and do a heartbeat RPC:
   ```
     public EndpointStateMachine.EndPointStates call() throws Exception {
       rpcEndpoint.lock();
   
       try {
         ...
         SCMHeartbeatResponseProto reponse = rpcEndpoint.getEndPoint()
             .sendHeartbeat(request);
         processResponse(reponse, datanodeDetailsProto);
         ...
       } catch (IOException ex) {
         ...
       } finally {
         rpcEndpoint.unlock();
       }
       return rpcEndpoint.getState();
     }
   ```
   
   Here is the problem:
   Say there is a slow Recon, two threads in thread pool. During the first heartbeat, one thread will acquire the lock and pending on the RPC with Recon, during the next heartbeat, the idle thread, after it finishes the job with SCM, it will finally blocked on acquiring endpoint lock of Recon, during the third heartbeat, there will no available threads in the thread pool.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org