You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2013/11/23 01:49:57 UTC
Spark driver behind NAT
Hello,
as far as i can tell, spark executors use akka to connect back to the
driver.
However, if driver is behind NAT, it becomes impossible since the tcp
connections are flowing from workers to the driver.
Is there any known way to set up spark clients behind NAT?
thanks. -Dmitriy
Re: Spark driver behind NAT
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
thank you, guys. much appreciated.
-d
On Sun, Nov 24, 2013 at 10:12 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Yup, it’s also important to have low latency between the drivers and the
> workers. If you plan to expose this to the outside (e.g. offer a shell
> interface), it would be better to write something on top.
>
> Matei
>
> On Nov 24, 2013, at 3:17 PM, Patrick Wendell <pw...@gmail.com> wrote:
>
> > Or more generally, it should be fully visible to the slave nodes
> > inside of the cluster.
> >
> > On Sun, Nov 24, 2013 at 3:16 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> >> Currently, no. The driver is a stateful component that is heavy-weight
> >> and should be run inside of the cluster.
> >>
> >> On Fri, Nov 22, 2013 at 4:49 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> >>> Hello,
> >>>
> >>> as far as i can tell, spark executors use akka to connect back to the
> >>> driver.
> >>>
> >>> However, if driver is behind NAT, it becomes impossible since the tcp
> >>> connections are flowing from workers to the driver.
> >>>
> >>> Is there any known way to set up spark clients behind NAT?
> >>>
> >>> thanks. -Dmitriy
>
>
Re: Spark driver behind NAT
Posted by Matei Zaharia <ma...@gmail.com>.
Yup, it’s also important to have low latency between the drivers and the workers. If you plan to expose this to the outside (e.g. offer a shell interface), it would be better to write something on top.
Matei
On Nov 24, 2013, at 3:17 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Or more generally, it should be fully visible to the slave nodes
> inside of the cluster.
>
> On Sun, Nov 24, 2013 at 3:16 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> Currently, no. The driver is a stateful component that is heavy-weight
>> and should be run inside of the cluster.
>>
>> On Fri, Nov 22, 2013 at 4:49 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>>> Hello,
>>>
>>> as far as i can tell, spark executors use akka to connect back to the
>>> driver.
>>>
>>> However, if driver is behind NAT, it becomes impossible since the tcp
>>> connections are flowing from workers to the driver.
>>>
>>> Is there any known way to set up spark clients behind NAT?
>>>
>>> thanks. -Dmitriy
Re: Spark driver behind NAT
Posted by Patrick Wendell <pw...@gmail.com>.
Or more generally, it should be fully visible to the slave nodes
inside of the cluster.
On Sun, Nov 24, 2013 at 3:16 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Currently, no. The driver is a stateful component that is heavy-weight
> and should be run inside of the cluster.
>
> On Fri, Nov 22, 2013 at 4:49 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>> Hello,
>>
>> as far as i can tell, spark executors use akka to connect back to the
>> driver.
>>
>> However, if driver is behind NAT, it becomes impossible since the tcp
>> connections are flowing from workers to the driver.
>>
>> Is there any known way to set up spark clients behind NAT?
>>
>> thanks. -Dmitriy
Re: Spark driver behind NAT
Posted by Patrick Wendell <pw...@gmail.com>.
Currently, no. The driver is a stateful component that is heavy-weight
and should be run inside of the cluster.
On Fri, Nov 22, 2013 at 4:49 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Hello,
>
> as far as i can tell, spark executors use akka to connect back to the
> driver.
>
> However, if driver is behind NAT, it becomes impossible since the tcp
> connections are flowing from workers to the driver.
>
> Is there any known way to set up spark clients behind NAT?
>
> thanks. -Dmitriy