[jira] [Created] (MESOS-9750) Agent V1 GET_STATE response may report a complete executor's tasks as non-terminal after a graceful agent shutdown

["Joseph Wu (JIRA)" <ji...@apache.org>]

Joseph Wu created MESOS-9750:
--------------------------------

             Summary: Agent V1 GET_STATE response may report a complete executor's tasks as non-terminal after a graceful agent shutdown
                 Key: MESOS-9750
                 URL: https://issues.apache.org/jira/browse/MESOS-9750
             Project: Mesos
          Issue Type: Bug
          Components: agent, executor
    Affects Versions: 1.7.0, 1.6.0, 1.8.0
            Reporter: Joseph Wu
            Assignee: Joseph Wu


When the following steps occur:
1) A graceful shutdown is initiated on the agent (i.e. SIGUSR1 or /master/machine/down).
2) The executor is sent a kill, and the agent counts down on {{executor_shutdown_grace_period}}.
3) The executor exits, before all terminal status updates reach the agent. This is more likely if {{executor_shutdown_grace_period}} passes.

This results in a completed executor, with non-terminal tasks (according to status updates).

When the agent starts back up, the completed executor will be recovered and shows up correctly  as a completed executor in {{/state}}.  However, if you fetch the V1 {{GET_STATE}} result, there will be an entry in {{launched_tasks}} even though nothing is running.
{code}
get_tasks {
  launched_tasks {
    name: "test-task"
    task_id {
      value: "dff5a155-47f1-4a71-9b92-30ca059ab456"
    }
    framework_id {
      value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
    }
    executor_id {
      value: "default"
    }
    agent_id {
      value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-S0"
    }
    state: TASK_RUNNING
    resources { ... }
    resources { ... }
    resources { ... }
    resources { ... }
    statuses {
      task_id {
        value: "dff5a155-47f1-4a71-9b92-30ca059ab456"
      }
      state: TASK_RUNNING
      agent_id {
        value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-S0"
      }
      timestamp: 1556674758.2175469
      executor_id {
        value: "default"
      }
      source: SOURCE_EXECUTOR
      uuid: "xPmn\234\236F&\235\\d\364\326\323\222\224"
      container_status { ... }
    }
  }
}
get_executors {
  completed_executors {
    executor_info {
      executor_id {
        value: "default"
      }
      command {
        value: ""
      }
      framework_id {
        value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
      }
    }
  }
}
get_frameworks {
  completed_frameworks {
    framework_info {
      user: "user"
      name: "default"
      id {
        value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
      }
      checkpoint: true
      hostname: "localhost"
      principal: "test-principal"
      capabilities {
        type: MULTI_ROLE
      }
      capabilities {
        type: RESERVATION_REFINEMENT
      }
      roles: "*"
    }
  }
}
{code}

This happens because we combine executors and completed executors when constructing the response.  The terminal task(s) with non-terminal updates appear under completed executors.
https://github.com/apache/mesos/blob/89c3dd95a421e14044bc91ceb1998ff4ae3883b4/src/slave/http.cpp#L1734-L1756



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)