You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Seokchan Ahn (JIRA)" <ji...@apache.org> on 2017/08/17 16:37:00 UTC

[jira] [Comment Edited] (REEF-836) Add Preemption API

    [ https://issues.apache.org/jira/browse/REEF-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130770#comment-16130770 ] 

Seokchan Ahn edited comment on REEF-836 at 8/17/17 4:36 PM:
------------------------------------------------------------

h1. Add Preemption API Design Proposal

h2. Goals
# Design clear Preemption REEF APIs.
# Implement new API for YARN and write tests that validates it.
# Implement the API on the local runtime for tests.

h2. Proposed Design

This design covers hard preemption only because soft preemption is technically not a preemption, but rather a hint.

h3. DriverConfiguration

{{EvaluatorPreemptedHandlers}} should be bound here.

{code:java}
public static final ConfigurationModule CONF = new DriverConfiguration().merge(DriverRuntimeConfiguration.CONF)
    ...
    .bindSetEntry(EvaluatorPreemptedHandlers.class, ON_EVALUATOR_PREEMPTED)
    ...
    .build();
{code}


h3. EvaluatorPreemptedHandlers

It will be called when an allocated evaluator is preempted. It will use {{DefaultEvaluatorPreemptedHandler}} as a default implementation.

h3. DefaultEvaluatorPreemptionHandler

It is a default handler for {{PreemptedEvaluator}}. For backward compatibility, it will simply redirect {{PreemptedEvaluator}} to {{FailedEvaluator}}.

{{EvaluatorFailedHandlers}} will be injected here and {{onNext(final PreemptedEvaluator preemptedEvaluator)}} will call each {{EvaluatorFailedHandler.onNext(preemptedEvaluator)}}.

{code:java}
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
    for (final EventHandler<FailedEvaluator> handler : evaluatorFailedHandlers) {
      handler.onNext(preemptedEvaluator);
    }
}
{code}


h3. PreemptedEvaluator

It should have the same interface with {{FailedEvaluator}} since preemption is a type of failure.
By extending {{FailedEvaluator}}, it can use the same APIs of {{FailedEvaluator}} and can be easily redirected to {{FailedEvaluator}}. PreemptedEvaluatorImpl would be almost same as FailedEvaluatorImpl.


h3. EvaluatorMessageDispatcher

{{EvaluatorMessageDispatcher}} is a central dispatcher for all evaluator related events. {{ServiceEvaluatorPreemptedEventHandlers}} should be registered here like below.

{code:java}
this.serviceDispatcher.register(PreemptedEvaluator.class, serviceEvaluatorPreemptedEventHandlers);
{code}



h2. Preemption API for YARN

h3. YarnContainerManager

{{YarnContainerManager}} handles container status event from YARN. 

When a {{PREEMPTED}} report comes, {{onContainerStatus(final ContainerStatus value)}} will update the status of a resource in a switch statement. Currently it sets status as KILLED for every kind of {{ContainerExitStatus}} (exit code 143 - termitated). It should be changed to handle each kind of {{ContainerExitStatus}}. exit code for PREEMPTED is -102.


h2. Preemption API on the local runtime for tests


h3. Step 1. Introduce a new local driver API for preemption. 

{code:java}
public interface PreemptionManager {
	void preempt(RunningTask task);
}
{code}


h3. Step 2. Add {{LocalPreemptionManager}}, an implementation of {{PreemptManager}} and bind it on {{LocalDriverConfiguration}}.

{{PreemptManager}} is a simple API for task preemption that contains preempt(RunningTask task), which closes task and call {{PreemptedEvaluatorHandler}}.

h3. Step 3. Preempt a {{RunningTask}} in {{RunningTaskHandler}}.

{code:java}
preemptionManager.preempt(runningTask);
{code}

I introduced task-level preemption API because a driver is independent of others in local runtime environment. But I am not sure of this approach. I would appreciate your feedback.


h2. Preemption Test

h3. YARN

# Configure 2 queues in YARN, queue A with 80% and queue B with 20% of total resources.
# Submit a job(preemptible) to B that consumes about 70% of total resources.
# Submit another job that consumes more than 70% of total resources. (Preemption is expected)
# Check if {{PreemptedEvaluator}} handler is invoked.


h3. Local Runtime

# Start a task.
# Let the PreemptionManager preempt the task.
# Check if {{PreemptedEvaluator}} handler is invoked.


h2. Proposed Use cases on REEF

h3. DriverConfiguration

{code:java}
final Configuration driverConfiguration = DriverConfiguration.CONF
        .set(DriverConfiguration.ON_EVALUATOR_PREEMPTED, <MyDriver>.EvaluatorPreemptedHandler.class)
        ...
        .build();
{code}


h3. PreemptedEvaluatorHandler

{code:java}
final class PreemptedEvaluatorHandler implements EventHandler<PreemptedEvaluator> {
  @Override
  public void onNext(final PreemptedEvaluator preemptedEvaluator) {
    // handle preemption
  }
}
{code}




was (Author: scan93):
h1. Add Preemption API Design Proposal

h2. Goals
# Design clear Preemption REEF APIs.
# Implement new API for YARN and write tests that validates it.
# Implement the API on the local runtime for tests.

h2. Proposed Design

This design covers hard preemption only because soft preemption is technically not a preemption, but rather a hint.

h3. DriverConfiguration

{{EvaluatorPreemptedHandlers}} should be bound here.

{code:java}
public static final ConfigurationModule CONF = new DriverConfiguration().merge(DriverRuntimeConfiguration.CONF)
    ...
    .bindSetEntry(EvaluatorPreemptedHandlers.class, ON_EVALUATOR_PREEMPTED)
    ...
    .build();
{code}


h3. EvaluatorPreemptedHandlers

It will be called when an allocated evaluator is preempted. It will use {{DefaultEvaluatorPreemptedHandler}} as a default implementation.

h3. DefaultEvaluatorPreemptionHandler

It is a default handler for {{PreemptedEvaluator}}. For backward compatibility, it will simply redirect {{PreemptedEvaluator}} to {{FailedEvaluator}}.

{{EvaluatorFailedHandlers}} will be injected here and {{onNext(final PreemptedEvaluator preemptedEvaluator)}} will call each {{EvaluatorFailedHandler.onNext(preemptedEvaluator)}}.

{code:java}
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
    for (final EventHandler<FailedEvaluator> handler : evaluatorFailedHandlers) {
      handler.onNext(preemptedEvaluator);
    }
}
{code}


h3. PreemptedEvaluator

It should have the same interface with {{FailedEvaluator}} since preemption is a type of failure.
By extending {{FailedEvaluator}}, it can use implemented APIs of {{FailedEvaluator}} and can be easily redirected to {{FailedEvaluator}}.


h3. EvaluatorMessageDispatcher

{{EvaluatorMessageDispatcher}} is a central dispatcher for all evaluator related events. {{ServiceEvaluatorPreemptedEventHandlers}} should be registered here like below.

{code:java}
this.serviceDispatcher.register(PreemptedEvaluator.class, serviceEvaluatorPreemptedEventHandlers);
{code}



h2. Preemption API for YARN

h3. YarnContainerManager

{{YarnContainerManager}} handles container status event from YARN. 

When a {{PREEMPTED}} report comes, {{onContainerStatus(final ContainerStatus value)}} will update the status of a resource in a switch statement. Currently it sets status as KILLED for every kind of {{ContainerExitStatus}} (exit code 143 - termitated). It should be changed to handle each kind of {{ContainerExitStatus}}. exit code for PREEMPTED is -102.


h2. Preemption API on the local runtime for tests


h3. Step 1. Introduce a new local driver API for preemption. 

{code:java}
public interface PreemptionManager {
	void preempt(RunningTask task);
}
{code}


h3. Step 2. Add {{LocalPreemptionManager}}, an implementation of {{PreemptManager}} and bind it on {{LocalDriverConfiguration}}.

{{PreemptManager}} is a simple API for task preemption that contains preempt(RunningTask task), which closes task and call {{PreemptedEvaluatorHandler}}.

h3. Step 3. Preempt a {{RunningTask}} in {{RunningTaskHandler}}.

{code:java}
preemptionManager.preempt(runningTask);
{code}

I introduced task-level preemption API because a driver is independent of others in local runtime environment. But I am not sure of this approach. I would appreciate your feedback.


h2. Preemption Test

h3. YARN

# Configure 2 queues in YARN, queue A with 80% and queue B with 20% of total resources.
# Submit a job(preemptible) to B that consumes about 70% of total resources.
# Submit another job that consumes more than 70% of total resources. (Preemption is expected)
# Check if {{PreemptedEvaluator}} handler is invoked.


h3. Local Runtime

# Start a task.
# Let the PreemptionManager preempt the task.
# Check if {{PreemptedEvaluator}} handler is invoked.


h2. Proposed Use cases on REEF

h3. DriverConfiguration

{code:java}
final Configuration driverConfiguration = DriverConfiguration.CONF
        .set(DriverConfiguration.ON_EVALUATOR_PREEMPTED, <MyDriver>.EvaluatorPreemptedHandler.class)
        ...
        .build();
{code}


h3. PreemptedEvaluatorHandler

{code:java}
final class PreemptedEvaluatorHandler implements EventHandler<PreemptedEvaluator> {
  @Override
  public void onNext(final PreemptedEvaluator preemptedEvaluator) {
    // handle preemption
  }
}
{code}



> Add Preemption API
> ------------------
>
>                 Key: REEF-836
>                 URL: https://issues.apache.org/jira/browse/REEF-836
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF, REEF Driver
>            Reporter: Mariia Mykhailova
>              Labels: gsoc2017
>         Attachments: Support Preemption on REEF Design Proposal.docx
>
>
> Umbrella JIRA for implementation of preemption in REEF. 
> Previously existed as [#690|https://github.com/Microsoft-CISL/REEF/issues/690].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)