SPARK_WORKER_PORT (standalone cluster)

[jay vyas <ja...@gmail.com>]

Hi spark !

What is the purpose of the randomly assigned SPARK_WORKER_PORT

from the documentation it sais to "join a cluster", but its not clear to me
how a random port could be used to communicate with other members of a
spark  pool.

This question might be grounded in my ignorance ... if so please just point
me to the right documentation if im mising something obvious :)

thanks !
-- 
jay vyas

Re: SPARK_WORKER_PORT (standalone cluster)

[jay vyas <ja...@gmail.com>]

Now I see the answer to this.

Spark slaves are start on random ports, and tell the master where they are.
then the master acknowledges them.

(worker logs)
Starting Spark worker xxxx:43282

(master logs)
Registering worker on xxxx:43282 with 8 cores, 16.5 GB RAM

Thus, the port is random because the slaves can be ephemeral.  Since the
master is fixed, though, a new slave can reconnect at any time.



On Mon, Jul 14, 2014 at 10:01 PM, jay vyas <ja...@gmail.com>
wrote:

> Hi spark !
>
> What is the purpose of the randomly assigned SPARK_WORKER_PORT
>
> from the documentation it sais to "join a cluster", but its not clear to
> me how a random port could be used to communicate with other members of a
> spark  pool.
>
> This question might be grounded in my ignorance ... if so please just
> point me to the right documentation if im mising something obvious :)
>
> thanks !
> --
> jay vyas
>



-- 
jay vyas