You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Gordon Hamilton <go...@wandisco.com> on 2013/08/20 17:53:22 UTC
JSON Endpoints
Hi guys,
I'm new to Spark, so apologies if I've missed something obvious.
I'm investigating the possibility of using Spark from an external UI, and see that Spark 0.7.0 introduced JSON endpoints into the UI which allows Spark's state to be queried. I see in MasterWebUI.scala that "/json" and "/app/json" map to renderJson functions, but when I hit the url's from a browser, I see "No service available for [/json]" and "No service available for [/app/json]" respectively.
Is there any documentation on how the end points work, or could someone point me in the right direction as to where I can find more information?
Many thanks in advance,
Gordon
--
*Register now for* Subversion & Git Live October 2013<http://www.wandisco.com/subversion-live-2013>
* **in** **Boston / San Francisco /** London*
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
If you are not the intended recipient, please notify us immediately and
destroy the message without disclosing its contents to anyone. Any
distribution, use or copying of this e-mail or the information it contains
by other than an intended recipient is unauthorized. The views and
opinions expressed in this e-mail message are the author's own and may not
reflect the views and opinions of WANdisco, unless the author is authorized
by WANdisco to express such views or opinions on its behalf. All email
sent to or from this address is subject to electronic storage and review by
WANdisco. Although WANdisco operates anti-virus programs, it does not
accept responsibility for any damage whatsoever caused by viruses being
passed.
Re: JSON Endpoints
Posted by Gordon Hamilton <op...@gmail.com>.
Hi Patrick,
I've got my calls working now. This is exactly the kind of thing I was looking for.
Thank you for all your help.
Best regards,
Gordon
On 21 Aug 2013, at 00:44, Patrick Wendell <pw...@gmail.com> wrote:
> Hey Gordon,
>
> We don't have a specification of the API right now (in 0.7.X this is a
> somewhat experimental feature). Each call returns a set of (k, v) pairs and
> the list of keys is static, so you can just do a function call in the
> browser to see what it returns and get the schema. The JSON protocol code
> is here:
>
> https://github.com/mesos/spark/blob/branch-0.7/core/src/main/scala/spark/deploy/JsonProtocol.scala
>
> For 0.8 this will be more cleanly specified.
>
> - Patrick
>
>
> On Tue, Aug 20, 2013 at 4:25 PM, Gordon Hamilton <
> gordon.hamilton@wandisco.com> wrote:
>
>> Hi Patrick, that's great. I think I've possibly started a build from
>> branch-0.7 but have been looking at branch-0.8 code. I'll try the formats
>> you have described in the morning.
>>
>> Is there any documentation that outlines all that I could expect to see
>> returned from the json calls?
>>
>> Thanks,
>> Gordon
>>
>> On Tuesday, August 20, 2013, Patrick Wendell wrote:
>>
>>> In 0.7.X these are actually of the format /app?format=json. In 0.8.X they
>>> will be the format you described here "/app/json". Does that work for
>> you?
>>>
>>> - Patrick
>>>
>>>
>>> On Tue, Aug 20, 2013 at 8:53 AM, Gordon Hamilton <
>>> gordon.hamilton@wandisco.com <javascript:;>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm new to Spark, so apologies if I've missed something obvious.
>>>>
>>>> I'm investigating the possibility of using Spark from an external UI,
>> and
>>>> see that Spark 0.7.0 introduced JSON endpoints into the UI which allows
>>>> Spark's state to be queried. I see in MasterWebUI.scala that "/json"
>> and
>>>> "/app/json" map to renderJson functions, but when I hit the url's from
>> a
>>>> browser, I see "No service available for [/json]" and "No service
>>> available
>>>> for [/app/json]" respectively.
>>>>
>>>> Is there any documentation on how the end points work, or could someone
>>>> point me in the right direction as to where I can find more
>> information?
>>>>
>>>> Many thanks in advance,
>>>> Gordon
>>>>
>>>>
>>>> --
>>>> *Register now for* Subversion & Git Live October 2013<
>>>> http://www.wandisco.com/subversion-live-2013>
>>>> * **in** **Boston / San Francisco /** London*
>>>>
>>>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>>> BE
>>>> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>>>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>>> privilege.
>>>> If you are not the intended recipient, please notify us immediately
>> and
>>>> destroy the message without disclosing its contents to anyone. Any
>>>> distribution, use or copying of this e-mail or the information it
>>> contains
>>>> by other than an intended recipient is unauthorized. The views and
>>>> opinions expressed in this e-mail message are the author's own and may
>>> not
>>>> reflect the views and opinions of WANdisco, unless the author is
>>> authorized
>>>> by WANdisco to express such views or opinions on its behalf. All email
>>>> sent to or from this address is subject to electronic storage and
>> review
>>> by
>>>> WANdisco. Although WANdisco operates anti-virus programs, it does not
>>>> accept responsibility for any damage whatsoever caused by viruses being
>>>> passed.
>>>>
>>>>
>>>
>>
>>
>> --
>> Gordon Hamilton I *Hadoop Development Team*
>>
>> WANdisco // *Non-Stop Data*
>>
>> e. gordon.hamilton@wandisco.com
>>
>> --
>> *Register now for* Subversion & Git Live October 2013<
>> http://www.wandisco.com/subversion-live-2013>
>> * **in** **Boston / San Francisco /** London*
>>
>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
>> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>> subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
>> If you are not the intended recipient, please notify us immediately and
>> destroy the message without disclosing its contents to anyone. Any
>> distribution, use or copying of this e-mail or the information it contains
>> by other than an intended recipient is unauthorized. The views and
>> opinions expressed in this e-mail message are the author's own and may not
>> reflect the views and opinions of WANdisco, unless the author is authorized
>> by WANdisco to express such views or opinions on its behalf. All email
>> sent to or from this address is subject to electronic storage and review by
>> WANdisco. Although WANdisco operates anti-virus programs, it does not
>> accept responsibility for any damage whatsoever caused by viruses being
>> passed.
>>
>>
Re: JSON Endpoints
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Gordon,
We don't have a specification of the API right now (in 0.7.X this is a
somewhat experimental feature). Each call returns a set of (k, v) pairs and
the list of keys is static, so you can just do a function call in the
browser to see what it returns and get the schema. The JSON protocol code
is here:
https://github.com/mesos/spark/blob/branch-0.7/core/src/main/scala/spark/deploy/JsonProtocol.scala
For 0.8 this will be more cleanly specified.
- Patrick
On Tue, Aug 20, 2013 at 4:25 PM, Gordon Hamilton <
gordon.hamilton@wandisco.com> wrote:
> Hi Patrick, that's great. I think I've possibly started a build from
> branch-0.7 but have been looking at branch-0.8 code. I'll try the formats
> you have described in the morning.
>
> Is there any documentation that outlines all that I could expect to see
> returned from the json calls?
>
> Thanks,
> Gordon
>
> On Tuesday, August 20, 2013, Patrick Wendell wrote:
>
> > In 0.7.X these are actually of the format /app?format=json. In 0.8.X they
> > will be the format you described here "/app/json". Does that work for
> you?
> >
> > - Patrick
> >
> >
> > On Tue, Aug 20, 2013 at 8:53 AM, Gordon Hamilton <
> > gordon.hamilton@wandisco.com <javascript:;>> wrote:
> >
> > > Hi guys,
> > >
> > > I'm new to Spark, so apologies if I've missed something obvious.
> > >
> > > I'm investigating the possibility of using Spark from an external UI,
> and
> > > see that Spark 0.7.0 introduced JSON endpoints into the UI which allows
> > > Spark's state to be queried. I see in MasterWebUI.scala that "/json"
> and
> > > "/app/json" map to renderJson functions, but when I hit the url's from
> a
> > > browser, I see "No service available for [/json]" and "No service
> > available
> > > for [/app/json]" respectively.
> > >
> > > Is there any documentation on how the end points work, or could someone
> > > point me in the right direction as to where I can find more
> information?
> > >
> > > Many thanks in advance,
> > > Gordon
> > >
> > >
> > > --
> > > *Register now for* Subversion & Git Live October 2013<
> > > http://www.wandisco.com/subversion-live-2013>
> > > * **in** **Boston / San Francisco /** London*
> > >
> > > THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
> > BE
> > > PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> > > subsidiaries, ("WANdisco") does not waive any confidentiality or
> > privilege.
> > > If you are not the intended recipient, please notify us immediately
> and
> > > destroy the message without disclosing its contents to anyone. Any
> > > distribution, use or copying of this e-mail or the information it
> > contains
> > > by other than an intended recipient is unauthorized. The views and
> > > opinions expressed in this e-mail message are the author's own and may
> > not
> > > reflect the views and opinions of WANdisco, unless the author is
> > authorized
> > > by WANdisco to express such views or opinions on its behalf. All email
> > > sent to or from this address is subject to electronic storage and
> review
> > by
> > > WANdisco. Although WANdisco operates anti-virus programs, it does not
> > > accept responsibility for any damage whatsoever caused by viruses being
> > > passed.
> > >
> > >
> >
>
>
> --
> Gordon Hamilton I *Hadoop Development Team*
>
> WANdisco // *Non-Stop Data*
>
> e. gordon.hamilton@wandisco.com
>
> --
> *Register now for* Subversion & Git Live October 2013<
> http://www.wandisco.com/subversion-live-2013>
> * **in** **Boston / San Francisco /** London*
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
> If you are not the intended recipient, please notify us immediately and
> destroy the message without disclosing its contents to anyone. Any
> distribution, use or copying of this e-mail or the information it contains
> by other than an intended recipient is unauthorized. The views and
> opinions expressed in this e-mail message are the author's own and may not
> reflect the views and opinions of WANdisco, unless the author is authorized
> by WANdisco to express such views or opinions on its behalf. All email
> sent to or from this address is subject to electronic storage and review by
> WANdisco. Although WANdisco operates anti-virus programs, it does not
> accept responsibility for any damage whatsoever caused by viruses being
> passed.
>
>
Re: JSON Endpoints
Posted by Gordon Hamilton <go...@wandisco.com>.
Hi Patrick, that's great. I think I've possibly started a build from
branch-0.7 but have been looking at branch-0.8 code. I'll try the formats
you have described in the morning.
Is there any documentation that outlines all that I could expect to see
returned from the json calls?
Thanks,
Gordon
On Tuesday, August 20, 2013, Patrick Wendell wrote:
> In 0.7.X these are actually of the format /app?format=json. In 0.8.X they
> will be the format you described here "/app/json". Does that work for you?
>
> - Patrick
>
>
> On Tue, Aug 20, 2013 at 8:53 AM, Gordon Hamilton <
> gordon.hamilton@wandisco.com <javascript:;>> wrote:
>
> > Hi guys,
> >
> > I'm new to Spark, so apologies if I've missed something obvious.
> >
> > I'm investigating the possibility of using Spark from an external UI, and
> > see that Spark 0.7.0 introduced JSON endpoints into the UI which allows
> > Spark's state to be queried. I see in MasterWebUI.scala that "/json" and
> > "/app/json" map to renderJson functions, but when I hit the url's from a
> > browser, I see "No service available for [/json]" and "No service
> available
> > for [/app/json]" respectively.
> >
> > Is there any documentation on how the end points work, or could someone
> > point me in the right direction as to where I can find more information?
> >
> > Many thanks in advance,
> > Gordon
> >
> >
> > --
> > *Register now for* Subversion & Git Live October 2013<
> > http://www.wandisco.com/subversion-live-2013>
> > * **in** **Boston / San Francisco /** London*
> >
> > THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
> BE
> > PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> > subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege.
> > If you are not the intended recipient, please notify us immediately and
> > destroy the message without disclosing its contents to anyone. Any
> > distribution, use or copying of this e-mail or the information it
> contains
> > by other than an intended recipient is unauthorized. The views and
> > opinions expressed in this e-mail message are the author's own and may
> not
> > reflect the views and opinions of WANdisco, unless the author is
> authorized
> > by WANdisco to express such views or opinions on its behalf. All email
> > sent to or from this address is subject to electronic storage and review
> by
> > WANdisco. Although WANdisco operates anti-virus programs, it does not
> > accept responsibility for any damage whatsoever caused by viruses being
> > passed.
> >
> >
>
--
Gordon Hamilton I *Hadoop Development Team*
WANdisco // *Non-Stop Data*
e. gordon.hamilton@wandisco.com
--
*Register now for* Subversion & Git Live October 2013<http://www.wandisco.com/subversion-live-2013>
* **in** **Boston / San Francisco /** London*
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
If you are not the intended recipient, please notify us immediately and
destroy the message without disclosing its contents to anyone. Any
distribution, use or copying of this e-mail or the information it contains
by other than an intended recipient is unauthorized. The views and
opinions expressed in this e-mail message are the author's own and may not
reflect the views and opinions of WANdisco, unless the author is authorized
by WANdisco to express such views or opinions on its behalf. All email
sent to or from this address is subject to electronic storage and review by
WANdisco. Although WANdisco operates anti-virus programs, it does not
accept responsibility for any damage whatsoever caused by viruses being
passed.
Re: JSON Endpoints
Posted by Patrick Wendell <pw...@gmail.com>.
In 0.7.X these are actually of the format /app?format=json. In 0.8.X they
will be the format you described here "/app/json". Does that work for you?
- Patrick
On Tue, Aug 20, 2013 at 8:53 AM, Gordon Hamilton <
gordon.hamilton@wandisco.com> wrote:
> Hi guys,
>
> I'm new to Spark, so apologies if I've missed something obvious.
>
> I'm investigating the possibility of using Spark from an external UI, and
> see that Spark 0.7.0 introduced JSON endpoints into the UI which allows
> Spark's state to be queried. I see in MasterWebUI.scala that "/json" and
> "/app/json" map to renderJson functions, but when I hit the url's from a
> browser, I see "No service available for [/json]" and "No service available
> for [/app/json]" respectively.
>
> Is there any documentation on how the end points work, or could someone
> point me in the right direction as to where I can find more information?
>
> Many thanks in advance,
> Gordon
>
>
> --
> *Register now for* Subversion & Git Live October 2013<
> http://www.wandisco.com/subversion-live-2013>
> * **in** **Boston / San Francisco /** London*
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
> If you are not the intended recipient, please notify us immediately and
> destroy the message without disclosing its contents to anyone. Any
> distribution, use or copying of this e-mail or the information it contains
> by other than an intended recipient is unauthorized. The views and
> opinions expressed in this e-mail message are the author's own and may not
> reflect the views and opinions of WANdisco, unless the author is authorized
> by WANdisco to express such views or opinions on its behalf. All email
> sent to or from this address is subject to electronic storage and review by
> WANdisco. Although WANdisco operates anti-virus programs, it does not
> accept responsibility for any damage whatsoever caused by viruses being
> passed.
>
>