You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by marco rocchi <ro...@studenti.uniroma1.it> on 2016/11/24 15:33:28 UTC

SparkUI via proxy

Hi,
I'm working with Apache Spark in order to develop my master thesis.I'm new
in spark and working with cluster. I searched through internet but I didn't
found a way to solve.
My problem is the following one: from my pc I can access to a master node
of a cluster only via proxy.
To connect to proxy and then to master node,I have to set up an ssh tunnel,
but from parctical point of view I have no idea of how in this way I can
interact with WebUI spark.
Anyone can help me?
Thanks in advance

Re: SparkUI via proxy

Posted by marco rocchi <ro...@studenti.uniroma1.it>.

Thanks to all, I solved the problem.
I'm sorry if the question was off topic, next time I'll post to
stackoverflow.
Thanks a lot

2016-11-25 17:19 GMT+01:00 marco rocchi <rocchi.1407763@studenti.uniroma1.it
>:

> Thanks for the helping.
> I've created my ssh tunnel at port 4040, and setted browser firefox SOCKS
> to localhost:4040.
> Now When I run a job I can read from INFO message: "SparkUI activated at
> http://192.168.1.204:4040". But if I open the browser and type local host
> or http://192.168.1.204:4040, webUI doesn't appear.
> Where I'm wrong?
> The question could be stupid, but I never worked with spark over a cluster
> :)
>
> Thanks
> Marco
>
> 2016-11-25 10:19 GMT+01:00 Ewan Leith <ew...@realitymine.com>:
>
>> This is more of a question for the spark user’s list, but if you look at
>> FoxyProxy and SSH tunnels it’ll get you going.
>>
>>
>>
>> These instructions from AWS for accessing EMR are a good start
>>
>>
>>
>> http://docs.aws.amazon.com/ElasticMapReduce/latest/Developer
>> Guide/emr-ssh-tunnel.html
>>
>>
>>
>> http://docs.aws.amazon.com/ElasticMapReduce/latest/Developer
>> Guide/emr-connect-master-node-proxy.html
>>
>>
>>
>> Ewan
>>
>>
>>
>> *From:* Georg Heiler [mailto:georg.kf.heiler@gmail.com]
>> *Sent:* 24 November 2016 16:41
>> *To:* marco rocchi <ro...@studenti.uniroma1.it>;
>> dev@spark.apache.org
>> *Subject:* Re: SparkUI via proxy
>>
>>
>>
>> Sehr Port forwarding will help you out.
>>
>> marco rocchi <ro...@studenti.uniroma1.it> schrieb am Do. 24.
>> Nov. 2016 um 16:33:
>>
>> Hi,
>>
>> I'm working with Apache Spark in order to develop my master thesis.I'm
>> new in spark and working with cluster. I searched through internet but I
>> didn't found a way to solve.
>>
>> My problem is the following one: from my pc I can access to a master node
>> of a cluster only via proxy.
>>
>> To connect to proxy and then to master node,I have to set up an ssh
>> tunnel, but from parctical point of view I have no idea of how in this way
>> I can interact with WebUI spark.
>>
>> Anyone can help me?
>>
>> Thanks in advance
>>
>>
>>
>> This email and any attachments to it may contain confidential information
>> and are intended solely for the addressee.
>>
>>
>> If you are not the intended recipient of this email or if you believe you
>> have received this email in error, please contact the sender and remove it
>> from your system.Do not use, copy or disclose the information contained in
>> this email or in any attachment.
>>
>> RealityMine Limited may monitor email traffic data including the content
>> of email for the purposes of security.
>>
>> RealityMine Limited is a company registered in England and Wales.
>> Registered number: 07920936 Registered office: Warren Bruce Court, Warren
>> Bruce Road, Trafford Park, Manchester M17 1LB
>>
>
>

Re: SparkUI via proxy

Posted by Ricardo Almeida <ri...@actnowib.com>.

Marco,

Depending on your configuration, maybe what you're looking for is:
localhost:4040

Check this two StackOverflow answers:
http://stackoverflow.com/questions/31460079/spark-ui-on-aws-emr
or similar questions. This is not a specific Spark issue.

Please check StackOverflow or post to User Mailing List, next time on this
type of question.


On 25 November 2016 at 17:19, marco rocchi <
rocchi.1407763@studenti.uniroma1.it> wrote:

> Thanks for the helping.
> I've created my ssh tunnel at port 4040, and setted browser firefox SOCKS
> to localhost:4040.
> Now When I run a job I can read from INFO message: "SparkUI activated at
> http://192.168.1.204:4040". But if I open the browser and type local host
> or http://192.168.1.204:4040, webUI doesn't appear.
> Where I'm wrong?
> The question could be stupid, but I never worked with spark over a cluster
> :)
>
> Thanks
> Marco
>
> 2016-11-25 10:19 GMT+01:00 Ewan Leith <ew...@realitymine.com>:
>
>> This is more of a question for the spark user’s list, but if you look at
>> FoxyProxy and SSH tunnels it’ll get you going.
>>
>>
>>
>> These instructions from AWS for accessing EMR are a good start
>>
>>
>>
>> http://docs.aws.amazon.com/ElasticMapReduce/latest/Developer
>> Guide/emr-ssh-tunnel.html
>>
>>
>>
>> http://docs.aws.amazon.com/ElasticMapReduce/latest/Developer
>> Guide/emr-connect-master-node-proxy.html
>>
>>
>>
>> Ewan
>>
>>
>>
>> *From:* Georg Heiler [mailto:georg.kf.heiler@gmail.com]
>> *Sent:* 24 November 2016 16:41
>> *To:* marco rocchi <ro...@studenti.uniroma1.it>;
>> dev@spark.apache.org
>> *Subject:* Re: SparkUI via proxy
>>
>>
>>
>> Sehr Port forwarding will help you out.
>>
>> marco rocchi <ro...@studenti.uniroma1.it> schrieb am Do. 24.
>> Nov. 2016 um 16:33:
>>
>> Hi,
>>
>> I'm working with Apache Spark in order to develop my master thesis.I'm
>> new in spark and working with cluster. I searched through internet but I
>> didn't found a way to solve.
>>
>> My problem is the following one: from my pc I can access to a master node
>> of a cluster only via proxy.
>>
>> To connect to proxy and then to master node,I have to set up an ssh
>> tunnel, but from parctical point of view I have no idea of how in this way
>> I can interact with WebUI spark.
>>
>> Anyone can help me?
>>
>> Thanks in advance
>>
>>
>>
>> This email and any attachments to it may contain confidential information
>> and are intended solely for the addressee.
>>
>>
>> If you are not the intended recipient of this email or if you believe you
>> have received this email in error, please contact the sender and remove it
>> from your system.Do not use, copy or disclose the information contained in
>> this email or in any attachment.
>>
>> RealityMine Limited may monitor email traffic data including the content
>> of email for the purposes of security.
>>
>> RealityMine Limited is a company registered in England and Wales.
>> Registered number: 07920936 Registered office: Warren Bruce Court, Warren
>> Bruce Road, Trafford Park, Manchester M17 1LB
>>
>
>

Re: SparkUI via proxy

Posted by marco rocchi <ro...@studenti.uniroma1.it>.

Thanks for the helping.
I've created my ssh tunnel at port 4040, and setted browser firefox SOCKS
to localhost:4040.
Now When I run a job I can read from INFO message: "SparkUI activated at
http://192.168.1.204:4040". But if I open the browser and type local host
or http://192.168.1.204:4040, webUI doesn't appear.
Where I'm wrong?
The question could be stupid, but I never worked with spark over a cluster
:)

Thanks
Marco

2016-11-25 10:19 GMT+01:00 Ewan Leith <ew...@realitymine.com>:

> This is more of a question for the spark user’s list, but if you look at
> FoxyProxy and SSH tunnels it’ll get you going.
>
>
>
> These instructions from AWS for accessing EMR are a good start
>
>
>
> http://docs.aws.amazon.com/ElasticMapReduce/latest/
> DeveloperGuide/emr-ssh-tunnel.html
>
>
>
> http://docs.aws.amazon.com/ElasticMapReduce/latest/
> DeveloperGuide/emr-connect-master-node-proxy.html
>
>
>
> Ewan
>
>
>
> *From:* Georg Heiler [mailto:georg.kf.heiler@gmail.com]
> *Sent:* 24 November 2016 16:41
> *To:* marco rocchi <ro...@studenti.uniroma1.it>;
> dev@spark.apache.org
> *Subject:* Re: SparkUI via proxy
>
>
>
> Sehr Port forwarding will help you out.
>
> marco rocchi <ro...@studenti.uniroma1.it> schrieb am Do. 24.
> Nov. 2016 um 16:33:
>
> Hi,
>
> I'm working with Apache Spark in order to develop my master thesis.I'm new
> in spark and working with cluster. I searched through internet but I didn't
> found a way to solve.
>
> My problem is the following one: from my pc I can access to a master node
> of a cluster only via proxy.
>
> To connect to proxy and then to master node,I have to set up an ssh
> tunnel, but from parctical point of view I have no idea of how in this way
> I can interact with WebUI spark.
>
> Anyone can help me?
>
> Thanks in advance
>
>
>
> This email and any attachments to it may contain confidential information
> and are intended solely for the addressee.
>
>
> If you are not the intended recipient of this email or if you believe you
> have received this email in error, please contact the sender and remove it
> from your system.Do not use, copy or disclose the information contained in
> this email or in any attachment.
>
> RealityMine Limited may monitor email traffic data including the content
> of email for the purposes of security.
>
> RealityMine Limited is a company registered in England and Wales.
> Registered number: 07920936 Registered office: Warren Bruce Court, Warren
> Bruce Road, Trafford Park, Manchester M17 1LB
>

RE: SparkUI via proxy

Posted by Ewan Leith <ew...@realitymine.com>.

This is more of a question for the spark user’s list, but if you look at FoxyProxy and SSH tunnels it’ll get you going.

These instructions from AWS for accessing EMR are a good start

http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-ssh-tunnel.html

http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-connect-master-node-proxy.html

Ewan

From: Georg Heiler [mailto:georg.kf.heiler@gmail.com]
Sent: 24 November 2016 16:41
To: marco rocchi <ro...@studenti.uniroma1.it>; dev@spark.apache.org
Subject: Re: SparkUI via proxy

Sehr Port forwarding will help you out.
marco rocchi <ro...@studenti.uniroma1.it>> schrieb am Do. 24. Nov. 2016 um 16:33:
Hi,
I'm working with Apache Spark in order to develop my master thesis.I'm new in spark and working with cluster. I searched through internet but I didn't found a way to solve.
My problem is the following one: from my pc I can access to a master node of a cluster only via proxy.
To connect to proxy and then to master node,I have to set up an ssh tunnel, but from parctical point of view I have no idea of how in this way I can interact with WebUI spark.
Anyone can help me?
Thanks in advance



This email and any attachments to it may contain confidential information and are intended solely for the addressee.



If you are not the intended recipient of this email or if you believe you have received this email in error, please contact the sender and remove it from your system.Do not use, copy or disclose the information contained in this email or in any attachment.

RealityMine Limited may monitor email traffic data including the content of email for the purposes of security.

RealityMine Limited is a company registered in England and Wales. Registered number: 07920936 Registered office: Warren Bruce Court, Warren Bruce Road, Trafford Park, Manchester M17 1LB

Re: SparkUI via proxy

Posted by Georg Heiler <ge...@gmail.com>.

Sehr Port forwarding will help you out.
marco rocchi <ro...@studenti.uniroma1.it> schrieb am Do. 24. Nov.
2016 um 16:33:

> Hi,
> I'm working with Apache Spark in order to develop my master thesis.I'm new
> in spark and working with cluster. I searched through internet but I didn't
> found a way to solve.
> My problem is the following one: from my pc I can access to a master node
> of a cluster only via proxy.
> To connect to proxy and then to master node,I have to set up an ssh
> tunnel, but from parctical point of view I have no idea of how in this way
> I can interact with WebUI spark.
> Anyone can help me?
> Thanks in advance
>