stand-alone deployMode=cluster problems

[Lanny Ripple <la...@spotright.com>]

We have a stand-alone spark 2.1.0 cluster of 9 nodes.  The master is a 2
cpu core with 4GB RAM.  Each worker node has 4 cpu cores and 8G of memory.
We run one executor per node with 2 cores and let spark grab the memory it
wants (6.9GB as listed in the UI).

We submit mobs with 7 cores-max, 2g driver, 2g per executor and cluster
deployMode.

We have a large process that runs from between 15 minutes to several
hours.  If we run a single job we find that the job runs in the "expected"
time.  If we submit two jobs at the same time we find that both jobs take
about twice as long to run as if we had run them alone.

Over the last few days we've torn into the logs and have looked at all
sorts of dstat and diamond graphs.  The hosts seem to be running without
issue.  Today we did a side-by-side compare from the UI of a single and
concurrent job and noticed that the driver node seems to attract tasks from
the other job.

Here is stage 123 when running by itself
[image: Inline image 1]

Here when running with another job.  The other driver was on host .152
[image: Inline image 2]

Stage 373 when run alone.  Note the use of the entire cluster.
[image: Inline image 5]

Here when run with the other jobs the other job's driver node becomes the
only one doing any work.
[image: Inline image 6]

Here is the Exectuor tab from the above slow job
[image: Inline image 3]

Here is the other job running at the same time (also about twice as slow as
if running alone)
[image: Inline image 4]

Above you can see from above each job's problem node is the node the other
job's driver is running on.

Any anyone knows what's going on here (and better yet has a fix) I'd love
to hear from you.

We've already tried using spark-jobserver.  We had a very tough time given
that we are so far ahead of the version it supports and have some Cassandra
requirements which make spark-jobserver deployment very painful.  We got it
working though and found about an 8% failure rate of submitted jobs.
Submitting as we are now dropped us to a failure rate of about 1 in 200
(which we basically consider random and don't worry about).

I guess for full disclosure I'll mention we are using spark's REST submit
API directly which is why we don't have an easy "run the driver on the
spark master" solution.  An easy fix would be the ability to specify which
node of the cluster the driver would run on in deployMode=cluster since
we'd just co-locate all the driver instances.  Sadly that doesn't seem to
be a thing.

Any anyone knows what's going on here (and better yet has a fix) I'd love
to hear from you.