You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Holden Karau <ho...@pigscanfly.ca> on 2021/08/20 23:01:40 UTC
Add option to Spark UI to proxy to the executors?
Hi Folks,
I'm wondering what people think about the idea of having the Spark UI
(optionally) act as a proxy to the executors? This could help with exec UI
access in some deployment environments.
Cheers,
Holden :)
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: Add option to Spark UI to proxy to the executors?
Posted by Ankit Gupta <in...@gmail.com>.
Hey Holden
I can probably help you with that. We should probably pass the same configs
to the application also,for the reverse proxy to work.
Try setting the spark.ui.reverseProxy=true while running your application.
I hope this will solve your issue.
Best Regards
Ankit Prakash Gupta
On Thu, 26 Aug, 2021, 2:48 am Holden Karau, <ho...@pigscanfly.ca> wrote:
> So I tried turning on the Spark exec UI proxy but it broke the Spark UI
> (in 3.1.2) and regardless of what URL I requested everything came back as
> text/html of the jobs page. Is anyone actively using this feature in prod?
>
> On Sun, Aug 22, 2021 at 5:58 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> Oh cool. I’ll have to dig down into why that’s not working with my K8s
>> deployment then.
>>
>> On Sat, Aug 21, 2021 at 11:54 PM Gengliang Wang <lt...@gmail.com> wrote:
>>
>>> Hi Holden,
>>>
>>> FYI there are already some related features in Spark:
>>>
>>> - Spark Master UI to reverse proxy Application and Workers UI
>>> <https://github.com/apache/spark/pull/13950>
>>> - Support Spark UI behind front-end reverse proxy using a path
>>> prefix Revert proxy URL <https://github.com/apache/spark/pull/29820>
>>>
>>> Not sure if they are helpful to you.
>>>
>>> On Sat, Aug 21, 2021 at 3:16 PM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Yes I can see your point.
>>>>
>>>> Will that work in kubernetes deployment?
>>>>
>>>> Mich
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, 21 Aug 2021 at 00:02, Holden Karau <ho...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> Hi Folks,
>>>>>
>>>>> I'm wondering what people think about the idea of having the Spark UI
>>>>> (optionally) act as a proxy to the executors? This could help with exec UI
>>>>> access in some deployment environments.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Holden :)
>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: Add option to Spark UI to proxy to the executors?
Posted by Holden Karau <ho...@pigscanfly.ca>.
So I tried turning on the Spark exec UI proxy but it broke the Spark UI (in
3.1.2) and regardless of what URL I requested everything came back as
text/html of the jobs page. Is anyone actively using this feature in prod?
On Sun, Aug 22, 2021 at 5:58 PM Holden Karau <ho...@pigscanfly.ca> wrote:
> Oh cool. I’ll have to dig down into why that’s not working with my K8s
> deployment then.
>
> On Sat, Aug 21, 2021 at 11:54 PM Gengliang Wang <lt...@gmail.com> wrote:
>
>> Hi Holden,
>>
>> FYI there are already some related features in Spark:
>>
>> - Spark Master UI to reverse proxy Application and Workers UI
>> <https://github.com/apache/spark/pull/13950>
>> - Support Spark UI behind front-end reverse proxy using a path prefix
>> Revert proxy URL <https://github.com/apache/spark/pull/29820>
>>
>> Not sure if they are helpful to you.
>>
>> On Sat, Aug 21, 2021 at 3:16 PM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Yes I can see your point.
>>>
>>> Will that work in kubernetes deployment?
>>>
>>> Mich
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sat, 21 Aug 2021 at 00:02, Holden Karau <ho...@pigscanfly.ca> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> I'm wondering what people think about the idea of having the Spark UI
>>>> (optionally) act as a proxy to the executors? This could help with exec UI
>>>> access in some deployment environments.
>>>>
>>>> Cheers,
>>>>
>>>> Holden :)
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: Add option to Spark UI to proxy to the executors?
Posted by Holden Karau <ho...@pigscanfly.ca>.
Oh cool. I’ll have to dig down into why that’s not working with my K8s
deployment then.
On Sat, Aug 21, 2021 at 11:54 PM Gengliang Wang <lt...@gmail.com> wrote:
> Hi Holden,
>
> FYI there are already some related features in Spark:
>
> - Spark Master UI to reverse proxy Application and Workers UI
> <https://github.com/apache/spark/pull/13950>
> - Support Spark UI behind front-end reverse proxy using a path prefix
> Revert proxy URL <https://github.com/apache/spark/pull/29820>
>
> Not sure if they are helpful to you.
>
> On Sat, Aug 21, 2021 at 3:16 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Yes I can see your point.
>>
>> Will that work in kubernetes deployment?
>>
>> Mich
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sat, 21 Aug 2021 at 00:02, Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> Hi Folks,
>>>
>>> I'm wondering what people think about the idea of having the Spark UI
>>> (optionally) act as a proxy to the executors? This could help with exec UI
>>> access in some deployment environments.
>>>
>>> Cheers,
>>>
>>> Holden :)
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: Add option to Spark UI to proxy to the executors?
Posted by Gengliang Wang <lt...@gmail.com>.
Hi Holden,
FYI there are already some related features in Spark:
- Spark Master UI to reverse proxy Application and Workers UI
<https://github.com/apache/spark/pull/13950>
- Support Spark UI behind front-end reverse proxy using a path prefix
Revert proxy URL <https://github.com/apache/spark/pull/29820>
Not sure if they are helpful to you.
On Sat, Aug 21, 2021 at 3:16 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Yes I can see your point.
>
> Will that work in kubernetes deployment?
>
> Mich
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 21 Aug 2021 at 00:02, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> Hi Folks,
>>
>> I'm wondering what people think about the idea of having the Spark UI
>> (optionally) act as a proxy to the executors? This could help with exec UI
>> access in some deployment environments.
>>
>> Cheers,
>>
>> Holden :)
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
Re: Add option to Spark UI to proxy to the executors?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Yes I can see your point.
Will that work in kubernetes deployment?
Mich
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 21 Aug 2021 at 00:02, Holden Karau <ho...@pigscanfly.ca> wrote:
> Hi Folks,
>
> I'm wondering what people think about the idea of having the Spark UI
> (optionally) act as a proxy to the executors? This could help with exec UI
> access in some deployment environments.
>
> Cheers,
>
> Holden :)
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>