How to start two Workers connected to two different masters

[onmstester onmstester <on...@zoho.com.INVALID>]

I have two java applications sharing the same spark cluster, the applications should be running on different servers. 

Based on my experience, if spark driver (inside java application) connects remotely to spark master (which is running on different node), then the response time to submit a job would be increased significantly in compare with the scenario that app&driver&master are running on the same node. 



So i need to have two spark master beside my applications.



 On the other hand i don't want to waste resources by dividing the spark cluster to two separate groups (two clusters that phisicaly separated ), therefore i want to start two slaves on each node of cluster, each (slave) connecting to one of masters and each could access all of the cores. The problem is that spark won't let me. First it said that you already have a worker, "stop it first". I set WORKER_INSTANCES to 2, but it creates two slave to one of the masters on first try!





Sent using https://www.zoho.com/mail/