Parallel spark jobs on standalone cluster

[Sarath Chandra <sa...@algofusiontech.com>]

Hi All,

I have a java program which submits a spark job to a standalone spark
cluster (2 nodes; 10 cores (6+4); 12GB (8+4)). This is being called by
another java program through ExecutorService and invokes it multiple times
with different set of arguments and parameters. I have set spark memory
usage to 3GB (tried with 2GB and 1GB also)

When I run this for 4 parallel jobs - first job finishes successfully,
second and third jobs start executing in parallel and never complete and
fourth job is waiting in queue. I've tried using "Fair Scheduler" but
didn't notice any change in the behavior. Also in the spark job submission
program I'm calling SparkContext.stop at the end of execution. Some times
all jobs fail with status as "Exited".

Please let me know what is going wrong and how to overcome the issue?

~Sarath