Whitelisting Spark ports in iptables

[Andrew Ash <an...@andrewash.com>]

Hi Spark list,

I'm looking to apply some iptables firewall rules to my spark cluster and
am not entirely sure what ports are required.  I didn't see any specific
documentation of what ports Spark requires, so compiled this (incomplete)
list.

*From* *To* *Port (default)* *Purpose* admin machine master 8080 master
webui admin machine worker 8081 worker webui admin machine application
4040application webuiworkermaster7077join clusterworkerworker?shuffle
datamasterworker?schedule jobs
Can someone help me out with the last couple ports and fill in any other
entries that might be missing?

Thanks!
Andrew

Re: Whitelisting Spark ports in iptables

[Mark Hamstra <ma...@clearstorydata.com>]

Problem is, though, that if a given port is found not to be available, then
Spark will try another; so your list is more a list of the first ports
Spark will try than it is of the ports Spark will use.


On Sun, Nov 24, 2013 at 5:06 PM, Andrew Ash <an...@andrewash.com> wrote:

> Hi Spark list,
>
> I'm looking to apply some iptables firewall rules to my spark cluster and
> am not entirely sure what ports are required.  I didn't see any specific
> documentation of what ports Spark requires, so compiled this (incomplete)
> list.
>
> *From* *To* *Port (default)* *Purpose* admin machine master 8080 master
> webui admin machine worker 8081 worker webui admin machine application4040application webuiworkermaster7077join clusterworkerworker?shuffle datamasterworker?schedule jobs
> Can someone help me out with the last couple ports and fill in any other
> entries that might be missing?
>
> Thanks!
> Andrew
>