Why would slot sharing be undesirable?

[le...@tutanota.com]

Hi,

i am deploying a 5 stage pipeline, with varying DOPs for experimental 
purposes. The benefits of slot sharing are clear. What is unclear to me is 
when slot sharing would be undesirable.

Since slot sharing enables bypassing of additional buffers (buffers between 
slots) as well as memory based optimizations, it surely is always beneficial. 
Furthermore, since all slots run within the same JVM, there also appears to 
be no advantage of disabling slot sharing in regard to simplifying JVM 
testing/debugging or for other isolation purposes.

Could you perhaps elaborate on when slot sharing may be undesirable?

Regards
Leon

Re: Why would slot sharing be undesirable?

[Márton Balassi <ba...@gmail.com>]

Hi Leon,

Basically you are trading away utilizing all of your resources in the
cluster to cut network IO.
With putting more and more tasks into the same slot you are pushing a job
from being potentially network bound to usually CPU bound, as the parallel
instances of tasks in the same slot sharing group are scheduled on the same
machine.

Have a look at the basic metrics of the working nodes of the cluster to
determine when to stop sharing.

Best,

Marton

On Sat, Jun 4, 2016 at 10:09 AM, <le...@tutanota.com> wrote:

> Hi,
>
> i am deploying a 5 stage pipeline, with varying DOPs for experimental
> purposes. The benefits of slot sharing are clear. What is unclear to me is
> when slot sharing would be undesirable.
>
> Since slot sharing enables bypassing of additional buffers (buffers
> between slots) as well as memory based optimizations, it surely is always
> beneficial. Furthermore, since all slots run within the same JVM, there
> also appears to be no advantage of disabling slot sharing in regard to
> simplifying JVM testing/debugging or for other isolation purposes.
>
> Could you perhaps elaborate on when slot sharing may be undesirable?
>
> Regards
> Leon
>