You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Seokchan Ahn (JIRA)" <ji...@apache.org> on 2017/08/17 16:37:00 UTC
[jira] [Comment Edited] (REEF-836) Add Preemption API
[ https://issues.apache.org/jira/browse/REEF-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130770#comment-16130770 ]
Seokchan Ahn edited comment on REEF-836 at 8/17/17 4:36 PM:
------------------------------------------------------------
h1. Add Preemption API Design Proposal
h2. Goals
# Design clear Preemption REEF APIs.
# Implement new API for YARN and write tests that validates it.
# Implement the API on the local runtime for tests.
h2. Proposed Design
This design covers hard preemption only because soft preemption is technically not a preemption, but rather a hint.
h3. DriverConfiguration
{{EvaluatorPreemptedHandlers}} should be bound here.
{code:java}
public static final ConfigurationModule CONF = new DriverConfiguration().merge(DriverRuntimeConfiguration.CONF)
...
.bindSetEntry(EvaluatorPreemptedHandlers.class, ON_EVALUATOR_PREEMPTED)
...
.build();
{code}
h3. EvaluatorPreemptedHandlers
It will be called when an allocated evaluator is preempted. It will use {{DefaultEvaluatorPreemptedHandler}} as a default implementation.
h3. DefaultEvaluatorPreemptionHandler
It is a default handler for {{PreemptedEvaluator}}. For backward compatibility, it will simply redirect {{PreemptedEvaluator}} to {{FailedEvaluator}}.
{{EvaluatorFailedHandlers}} will be injected here and {{onNext(final PreemptedEvaluator preemptedEvaluator)}} will call each {{EvaluatorFailedHandler.onNext(preemptedEvaluator)}}.
{code:java}
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
for (final EventHandler<FailedEvaluator> handler : evaluatorFailedHandlers) {
handler.onNext(preemptedEvaluator);
}
}
{code}
h3. PreemptedEvaluator
It should have the same interface with {{FailedEvaluator}} since preemption is a type of failure.
By extending {{FailedEvaluator}}, it can use the same APIs of {{FailedEvaluator}} and can be easily redirected to {{FailedEvaluator}}. PreemptedEvaluatorImpl would be almost same as FailedEvaluatorImpl.
h3. EvaluatorMessageDispatcher
{{EvaluatorMessageDispatcher}} is a central dispatcher for all evaluator related events. {{ServiceEvaluatorPreemptedEventHandlers}} should be registered here like below.
{code:java}
this.serviceDispatcher.register(PreemptedEvaluator.class, serviceEvaluatorPreemptedEventHandlers);
{code}
h2. Preemption API for YARN
h3. YarnContainerManager
{{YarnContainerManager}} handles container status event from YARN.
When a {{PREEMPTED}} report comes, {{onContainerStatus(final ContainerStatus value)}} will update the status of a resource in a switch statement. Currently it sets status as KILLED for every kind of {{ContainerExitStatus}} (exit code 143 - termitated). It should be changed to handle each kind of {{ContainerExitStatus}}. exit code for PREEMPTED is -102.
h2. Preemption API on the local runtime for tests
h3. Step 1. Introduce a new local driver API for preemption.
{code:java}
public interface PreemptionManager {
void preempt(RunningTask task);
}
{code}
h3. Step 2. Add {{LocalPreemptionManager}}, an implementation of {{PreemptManager}} and bind it on {{LocalDriverConfiguration}}.
{{PreemptManager}} is a simple API for task preemption that contains preempt(RunningTask task), which closes task and call {{PreemptedEvaluatorHandler}}.
h3. Step 3. Preempt a {{RunningTask}} in {{RunningTaskHandler}}.
{code:java}
preemptionManager.preempt(runningTask);
{code}
I introduced task-level preemption API because a driver is independent of others in local runtime environment. But I am not sure of this approach. I would appreciate your feedback.
h2. Preemption Test
h3. YARN
# Configure 2 queues in YARN, queue A with 80% and queue B with 20% of total resources.
# Submit a job(preemptible) to B that consumes about 70% of total resources.
# Submit another job that consumes more than 70% of total resources. (Preemption is expected)
# Check if {{PreemptedEvaluator}} handler is invoked.
h3. Local Runtime
# Start a task.
# Let the PreemptionManager preempt the task.
# Check if {{PreemptedEvaluator}} handler is invoked.
h2. Proposed Use cases on REEF
h3. DriverConfiguration
{code:java}
final Configuration driverConfiguration = DriverConfiguration.CONF
.set(DriverConfiguration.ON_EVALUATOR_PREEMPTED, <MyDriver>.EvaluatorPreemptedHandler.class)
...
.build();
{code}
h3. PreemptedEvaluatorHandler
{code:java}
final class PreemptedEvaluatorHandler implements EventHandler<PreemptedEvaluator> {
@Override
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
// handle preemption
}
}
{code}
was (Author: scan93):
h1. Add Preemption API Design Proposal
h2. Goals
# Design clear Preemption REEF APIs.
# Implement new API for YARN and write tests that validates it.
# Implement the API on the local runtime for tests.
h2. Proposed Design
This design covers hard preemption only because soft preemption is technically not a preemption, but rather a hint.
h3. DriverConfiguration
{{EvaluatorPreemptedHandlers}} should be bound here.
{code:java}
public static final ConfigurationModule CONF = new DriverConfiguration().merge(DriverRuntimeConfiguration.CONF)
...
.bindSetEntry(EvaluatorPreemptedHandlers.class, ON_EVALUATOR_PREEMPTED)
...
.build();
{code}
h3. EvaluatorPreemptedHandlers
It will be called when an allocated evaluator is preempted. It will use {{DefaultEvaluatorPreemptedHandler}} as a default implementation.
h3. DefaultEvaluatorPreemptionHandler
It is a default handler for {{PreemptedEvaluator}}. For backward compatibility, it will simply redirect {{PreemptedEvaluator}} to {{FailedEvaluator}}.
{{EvaluatorFailedHandlers}} will be injected here and {{onNext(final PreemptedEvaluator preemptedEvaluator)}} will call each {{EvaluatorFailedHandler.onNext(preemptedEvaluator)}}.
{code:java}
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
for (final EventHandler<FailedEvaluator> handler : evaluatorFailedHandlers) {
handler.onNext(preemptedEvaluator);
}
}
{code}
h3. PreemptedEvaluator
It should have the same interface with {{FailedEvaluator}} since preemption is a type of failure.
By extending {{FailedEvaluator}}, it can use implemented APIs of {{FailedEvaluator}} and can be easily redirected to {{FailedEvaluator}}.
h3. EvaluatorMessageDispatcher
{{EvaluatorMessageDispatcher}} is a central dispatcher for all evaluator related events. {{ServiceEvaluatorPreemptedEventHandlers}} should be registered here like below.
{code:java}
this.serviceDispatcher.register(PreemptedEvaluator.class, serviceEvaluatorPreemptedEventHandlers);
{code}
h2. Preemption API for YARN
h3. YarnContainerManager
{{YarnContainerManager}} handles container status event from YARN.
When a {{PREEMPTED}} report comes, {{onContainerStatus(final ContainerStatus value)}} will update the status of a resource in a switch statement. Currently it sets status as KILLED for every kind of {{ContainerExitStatus}} (exit code 143 - termitated). It should be changed to handle each kind of {{ContainerExitStatus}}. exit code for PREEMPTED is -102.
h2. Preemption API on the local runtime for tests
h3. Step 1. Introduce a new local driver API for preemption.
{code:java}
public interface PreemptionManager {
void preempt(RunningTask task);
}
{code}
h3. Step 2. Add {{LocalPreemptionManager}}, an implementation of {{PreemptManager}} and bind it on {{LocalDriverConfiguration}}.
{{PreemptManager}} is a simple API for task preemption that contains preempt(RunningTask task), which closes task and call {{PreemptedEvaluatorHandler}}.
h3. Step 3. Preempt a {{RunningTask}} in {{RunningTaskHandler}}.
{code:java}
preemptionManager.preempt(runningTask);
{code}
I introduced task-level preemption API because a driver is independent of others in local runtime environment. But I am not sure of this approach. I would appreciate your feedback.
h2. Preemption Test
h3. YARN
# Configure 2 queues in YARN, queue A with 80% and queue B with 20% of total resources.
# Submit a job(preemptible) to B that consumes about 70% of total resources.
# Submit another job that consumes more than 70% of total resources. (Preemption is expected)
# Check if {{PreemptedEvaluator}} handler is invoked.
h3. Local Runtime
# Start a task.
# Let the PreemptionManager preempt the task.
# Check if {{PreemptedEvaluator}} handler is invoked.
h2. Proposed Use cases on REEF
h3. DriverConfiguration
{code:java}
final Configuration driverConfiguration = DriverConfiguration.CONF
.set(DriverConfiguration.ON_EVALUATOR_PREEMPTED, <MyDriver>.EvaluatorPreemptedHandler.class)
...
.build();
{code}
h3. PreemptedEvaluatorHandler
{code:java}
final class PreemptedEvaluatorHandler implements EventHandler<PreemptedEvaluator> {
@Override
public void onNext(final PreemptedEvaluator preemptedEvaluator) {
// handle preemption
}
}
{code}
> Add Preemption API
> ------------------
>
> Key: REEF-836
> URL: https://issues.apache.org/jira/browse/REEF-836
> Project: REEF
> Issue Type: New Feature
> Components: REEF, REEF Driver
> Reporter: Mariia Mykhailova
> Labels: gsoc2017
> Attachments: Support Preemption on REEF Design Proposal.docx
>
>
> Umbrella JIRA for implementation of preemption in REEF.
> Previously existed as [#690|https://github.com/Microsoft-CISL/REEF/issues/690].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)