[DISCUSSION] Preemption design

[Wilfred Spiegelenburg <wi...@apache.org>]

Hi,

Preemption has been a much requested feature. Over the last weeks Craig and
I have worked on getting ready for a new implementation. Craig has removed
the old code so we can start with a clean slate.

I wrote a new design for preemption [1] based on the experience we have
with YARN preemption, specially with the FairScheduler. The focus of the
design is solely on inter queue or cross queue preemption.
Intra queue (within the queue) has been left out for now. A number of the
possible intra queue use cases can be achieved via inter queue preemption
with a slight change in queue configuration. Intra queue preemption could
always be added in the future if we see a need. I do think that keeping it
as simple as possible to start with is a much better approach.

Wilfred

[1]
https://docs.google.com/document/d/1j1Zc-YYHL5NlrC1grcBI00a32Dr5Q0v0f5tBrniRx7o/edit?usp=sharing