You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by John Omernik <jo...@omernik.com> on 2016/04/04 02:24:20 UTC
Spark Job Server
Hello all,
Has there been any discussion on using the Spark Job Server with Zeppelin?
https://github.com/spark-jobserver/spark-jobserver
My thought is this: Trying to get get Zeppelin Running in Docker and then
able to interact with Spark on Mesos is a challenge. It requires us to
include a lot of heavy native mesos libraries to the Docker Container.
The job server seems to be a good way to consolidate some of that (it can
interact with Mesos) and then have the interaction between Zeppelin and Job
Server be based on a Rest API. I am a bit over my head with some of the
specifics, but thought I'd ask here.
Other than something like the job server, what are other thoughts for
interacting with Spark on Mesos while running Zeppelin in a Docker
container? Anyone had any success? Care to share notes?
John
Re: Spark Job Server
Posted by John Omernik <jo...@omernik.com>.
Interesting. Has https://github.com/spark-jobserver/spark-jobserver been
compared to toree and Livy? What is the licensing on Livy, could that
cause issues down stream? (I am new to some of these things).
John
On Mon, Apr 4, 2016 at 6:35 AM, moon soo Lee <mo...@apache.org> wrote:
> Hi John,
>
> A proposal [1] attached in ZEPPELIN-773 [2] shows spark interpreter based
> on livy [3]. I think this is similar to what you think.
> Please take a look.
>
> Thanks,
> moon
>
> [1]
> https://docs.google.com/document/d/1AVGcviyVqWmmbHJmkgUo76ZDSwWAMjwHxmKBhZdAav4/edit#heading=h.5uzzc48tw5d8
> [2] https://issues.apache.org/jira/browse/ZEPPELIN-773
> [3] https://github.com/cloudera/livy
>
>
>
> On Mon, Apr 4, 2016 at 9:24 AM John Omernik <jo...@omernik.com> wrote:
>
>> Hello all,
>>
>> Has there been any discussion on using the Spark Job Server with
>> Zeppelin? https://github.com/spark-jobserver/spark-jobserver
>>
>> My thought is this: Trying to get get Zeppelin Running in Docker and then
>> able to interact with Spark on Mesos is a challenge. It requires us to
>> include a lot of heavy native mesos libraries to the Docker Container.
>> The job server seems to be a good way to consolidate some of that (it can
>> interact with Mesos) and then have the interaction between Zeppelin and Job
>> Server be based on a Rest API. I am a bit over my head with some of the
>> specifics, but thought I'd ask here.
>>
>> Other than something like the job server, what are other thoughts for
>> interacting with Spark on Mesos while running Zeppelin in a Docker
>> container? Anyone had any success? Care to share notes?
>>
>> John
>>
>
Re: Spark Job Server
Posted by moon soo Lee <mo...@apache.org>.
Hi John,
A proposal [1] attached in ZEPPELIN-773 [2] shows spark interpreter based
on livy [3]. I think this is similar to what you think.
Please take a look.
Thanks,
moon
[1]
https://docs.google.com/document/d/1AVGcviyVqWmmbHJmkgUo76ZDSwWAMjwHxmKBhZdAav4/edit#heading=h.5uzzc48tw5d8
[2] https://issues.apache.org/jira/browse/ZEPPELIN-773
[3] https://github.com/cloudera/livy
On Mon, Apr 4, 2016 at 9:24 AM John Omernik <jo...@omernik.com> wrote:
> Hello all,
>
> Has there been any discussion on using the Spark Job Server with Zeppelin?
> https://github.com/spark-jobserver/spark-jobserver
>
> My thought is this: Trying to get get Zeppelin Running in Docker and then
> able to interact with Spark on Mesos is a challenge. It requires us to
> include a lot of heavy native mesos libraries to the Docker Container.
> The job server seems to be a good way to consolidate some of that (it can
> interact with Mesos) and then have the interaction between Zeppelin and Job
> Server be based on a Rest API. I am a bit over my head with some of the
> specifics, but thought I'd ask here.
>
> Other than something like the job server, what are other thoughts for
> interacting with Spark on Mesos while running Zeppelin in a Docker
> container? Anyone had any success? Care to share notes?
>
> John
>