exposing spark through a web service

[Philip Ogren <ph...@oracle.com>]

Hi Spark Community,

I would like to expose my spark application/libraries via a web service 
in order to launch jobs, interact with users, etc.  I'm sure there are 
100's of ways to think about doing this each with a variety of 
technology stacks that could be applied.  So, I know there is no right 
answer here.  But, I am curious to know if anyone has a fairly general 
solution that they are happy with that they would recommend or if there 
is any growing consensus in this community towards a solution or family 
of solutions that align with the "Spark way" of doing things.  Any 
advice or thoughts are appreciated!

Thanks,
Philip

Re: exposing spark through a web service

[Romain Rigaux <ro...@gmail.com>]

We added an alpha version of a Spark App in Hue:
https://dl.dropboxusercontent.com/u/730827/cloudera/spark-app.png

However, we are using Oozie and the Spark shells so it means RDD are not
re-used and there is no interactivity.

We are currently doing a test about switching to the new REST Job Server
from pull/222 as there is good ways to re-use contexts. However it would be
Scala only and jars would need to be uploaded (so no shell-like
interactivity). We would welcome Java, Python and being able to send
scripts directly to the Job Server :)

Romain


On Fri, Dec 13, 2013 at 10:25 AM, Patrick Wendell <pw...@gmail.com>wrote:

> Hey Philip,
>
> To elaborate a bit, this is a proposed patch for integrating something
> like a restful server into Spark. If you wanted to take a look at the
> documentation in that patch and comment as to whether it would partially or
> fully solve your use-case that would be great.
>
> - Patrick
>
>
> On Fri, Dec 13, 2013 at 9:37 AM, Mark Hamstra <ma...@clearstorydata.com>wrote:
>
>> https://github.com/apache/incubator-spark/pull/222
>>
>>
>> On Fri, Dec 13, 2013 at 8:36 AM, Philip Ogren <ph...@oracle.com>wrote:
>>
>>> Hi Spark Community,
>>>
>>> I would like to expose my spark application/libraries via a web service
>>> in order to launch jobs, interact with users, etc.  I'm sure there are
>>> 100's of ways to think about doing this each with a variety of technology
>>> stacks that could be applied.  So, I know there is no right answer here.
>>>  But, I am curious to know if anyone has a fairly general solution that
>>> they are happy with that they would recommend or if there is any growing
>>> consensus in this community towards a solution or family of solutions that
>>> align with the "Spark way" of doing things.  Any advice or thoughts are
>>> appreciated!
>>>
>>> Thanks,
>>> Philip
>>>
>>
>>
>

Re: exposing spark through a web service

[Patrick Wendell <pw...@gmail.com>]

Hey Philip,

To elaborate a bit, this is a proposed patch for integrating something like
a restful server into Spark. If you wanted to take a look at the
documentation in that patch and comment as to whether it would partially or
fully solve your use-case that would be great.

- Patrick


On Fri, Dec 13, 2013 at 9:37 AM, Mark Hamstra <ma...@clearstorydata.com>wrote:

> https://github.com/apache/incubator-spark/pull/222
>
>
> On Fri, Dec 13, 2013 at 8:36 AM, Philip Ogren <ph...@oracle.com>wrote:
>
>> Hi Spark Community,
>>
>> I would like to expose my spark application/libraries via a web service
>> in order to launch jobs, interact with users, etc.  I'm sure there are
>> 100's of ways to think about doing this each with a variety of technology
>> stacks that could be applied.  So, I know there is no right answer here.
>>  But, I am curious to know if anyone has a fairly general solution that
>> they are happy with that they would recommend or if there is any growing
>> consensus in this community towards a solution or family of solutions that
>> align with the "Spark way" of doing things.  Any advice or thoughts are
>> appreciated!
>>
>> Thanks,
>> Philip
>>
>
>

Re: exposing spark through a web service

[Mark Hamstra <ma...@clearstorydata.com>]

https://github.com/apache/incubator-spark/pull/222


On Fri, Dec 13, 2013 at 8:36 AM, Philip Ogren <ph...@oracle.com>wrote:

> Hi Spark Community,
>
> I would like to expose my spark application/libraries via a web service in
> order to launch jobs, interact with users, etc.  I'm sure there are 100's
> of ways to think about doing this each with a variety of technology stacks
> that could be applied.  So, I know there is no right answer here.  But, I
> am curious to know if anyone has a fairly general solution that they are
> happy with that they would recommend or if there is any growing consensus
> in this community towards a solution or family of solutions that align with
> the "Spark way" of doing things.  Any advice or thoughts are appreciated!
>
> Thanks,
> Philip
>