accessing spark ui over an ssh tunnel

[Stephen Haberman <st...@gmail.com>]

Hey,

The new Spark UI looks awesome. Unfortunately, it's hard for me to use
in a real browser (on my desktop, vs. lynx on the cluster), since our
(EMR) clusters are basically only accessible via SSH.

I can port forward the master:8080 web UI port locally, and it looks
great, but I can't follow any links through to the slaves. The links
technically use the public AWS host names, but we don't have those
ports opened in the security group.

Two things come to mind:

1) Maybe the entire UI could be rendered on the master? Either based on
the master's data or making actor/JSON/whatever calls to the slaves.

2) Somehow proxy links through the master to each slave. Like
master:8080/slave/<id>/<some-path> ends up proxied to the
slave:8081/<some-path>, and the master just sends the HTML as-is back
to the client.

I'm not very familiar with the web UI, so I'm not sure which would be
easier. I also don't mean to bikeshed on the really great UI.

But, at least with our current setup, this would demonstrably increase
the usability for me.

Is this something that you guys have considered? Do you have any
thoughts on it? WIP? Open to contributions?

Thanks!

- Stephen

Re: accessing spark ui over an ssh tunnel

[Jey Kottalam <je...@cs.berkeley.edu>]

Minor nitpick: a proxy created using the "-D" flag should be
configured as a SOCKS proxy, not an HTTP proxy.

On Fri, Oct 25, 2013 at 9:42 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Hey Stephen,
>
> SSH actually supports creating a HTTP proxy through the -D flag. Take a look at the -D option on our spark-ec2 script for example, which just exposes the -D option of ssh. With this feature you can do stuff like ssh -D 8088 <host> and then configure localhost:8088 as a proxy in your web browser, and browse any page accessible from the machine you've SSHed into.
>
> Matei
>
> On Oct 25, 2013, at 7:40 PM, Stephen Haberman <st...@gmail.com> wrote:
>
>> Hey,
>>
>> The new Spark UI looks awesome. Unfortunately, it's hard for me to use
>> in a real browser (on my desktop, vs. lynx on the cluster), since our
>> (EMR) clusters are basically only accessible via SSH.
>>
>> I can port forward the master:8080 web UI port locally, and it looks
>> great, but I can't follow any links through to the slaves. The links
>> technically use the public AWS host names, but we don't have those
>> ports opened in the security group.
>>
>> Two things come to mind:
>>
>> 1) Maybe the entire UI could be rendered on the master? Either based on
>> the master's data or making actor/JSON/whatever calls to the slaves.
>>
>> 2) Somehow proxy links through the master to each slave. Like
>> master:8080/slave/<id>/<some-path> ends up proxied to the
>> slave:8081/<some-path>, and the master just sends the HTML as-is back
>> to the client.
>>
>> I'm not very familiar with the web UI, so I'm not sure which would be
>> easier. I also don't mean to bikeshed on the really great UI.
>>
>> But, at least with our current setup, this would demonstrably increase
>> the usability for me.
>>
>> Is this something that you guys have considered? Do you have any
>> thoughts on it? WIP? Open to contributions?
>>
>> Thanks!
>>
>> - Stephen
>

Re: accessing spark ui over an ssh tunnel

[Stephen Haberman <st...@gmail.com>]

Hi Matei,

> SSH actually supports creating a HTTP proxy through the -D flag.

Wow, nice. I hadn't seen that. I'll try that out.

As a nit, switching my browser over to use a proxy is slightly
annoying, but I'll do that for now. Maybe I can setup a dedicated
"spark proxy" user/profile in Chrome or something.

Thanks!

- Stephen

Re: accessing spark ui over an ssh tunnel

[Matei Zaharia <ma...@gmail.com>]

Hey Stephen,

SSH actually supports creating a HTTP proxy through the -D flag. Take a look at the -D option on our spark-ec2 script for example, which just exposes the -D option of ssh. With this feature you can do stuff like ssh -D 8088 <host> and then configure localhost:8088 as a proxy in your web browser, and browse any page accessible from the machine you've SSHed into.

Matei

On Oct 25, 2013, at 7:40 PM, Stephen Haberman <st...@gmail.com> wrote:

> Hey,
> 
> The new Spark UI looks awesome. Unfortunately, it's hard for me to use
> in a real browser (on my desktop, vs. lynx on the cluster), since our
> (EMR) clusters are basically only accessible via SSH.
> 
> I can port forward the master:8080 web UI port locally, and it looks
> great, but I can't follow any links through to the slaves. The links
> technically use the public AWS host names, but we don't have those
> ports opened in the security group.
> 
> Two things come to mind:
> 
> 1) Maybe the entire UI could be rendered on the master? Either based on
> the master's data or making actor/JSON/whatever calls to the slaves.
> 
> 2) Somehow proxy links through the master to each slave. Like
> master:8080/slave/<id>/<some-path> ends up proxied to the
> slave:8081/<some-path>, and the master just sends the HTML as-is back
> to the client.
> 
> I'm not very familiar with the web UI, so I'm not sure which would be
> easier. I also don't mean to bikeshed on the really great UI.
> 
> But, at least with our current setup, this would demonstrably increase
> the usability for me.
> 
> Is this something that you guys have considered? Do you have any
> thoughts on it? WIP? Open to contributions?
> 
> Thanks!
> 
> - Stephen