You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Harsh Rathi <co...@gmail.com> on 2015/05/15 15:02:11 UTC
Invoking and passing SparkContext to Zeppelin
Hi,
I am new to Zeppelin.
I want to share Spark Context with Zeppelin.
I want to invoke zeppelin and pass Spark Context to it so that whenever my
app is started, zeppelin gets started and my spark context is used by
Zeppelin. Is there any way to do it ?
Harsh Rathi
Re: Invoking and passing SparkContext to Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
RemoteInterpreter was introduced to avoid jar dependency conflict,
stdin/stdout/stderr conflict between interpreters. It runs interpreter in a
separate JVM process to avoid such conflicts. That was the main purpose of
RemoteInterpreter. RemoteInterpreter process created/destroyed from
ZeppelinServer.
For communication between ZeppelinServer and RemoteInterpreter, it uses
thrift protocol. So, basically, RemoteInterpreter is not limited to run in
the same machine that ZeppelinServer runs. It could run on remote machine.
But Zeppelin does not have features to connect already running
RemoteInterpreter process. (like GUI, options, etc).
Features like restarting ZeppelinServer without restarting
RemoteInterpreter, Bind RemoteInterpreter in remote machine to
ZeppelinServer was discussed from the beginning but simply we're not there
yet.
Zeppelin supports define properties(arguments) in your custom interpreter.
When properties for your interpreters are defined, users are able to use
edit them in 'Interpreter' menu in Zeppelin GUI.
Here's some source code you can check.
You can define property using InterpreterPropertyBuilder (eg.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L93
)
You can retrieve property value using getProperty() (eg.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L245
)
Thanks,
moon
On Tue, May 19, 2015 at 2:09 PM Harsh Rathi <co...@gmail.com>
wrote:
> Hi,
>
> I think I am getting confused here.
> So, what is the purpose of remote interpreter ?
> How remote server will be used if it can't connect to Zeppelin Server ?
>
>
> Is there any way I can invoke and pass custom arguments at run time to my
> Custom Interpreter which may run alongwith Zeppelin Server ?
>
> Thanks
> Harsh
>
> On Tue, May 19, 2015 at 4:54 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi,
>>
>> One completely difference possible approach is, you can run your app
>> inside of Zeppelin. Something like
>>
>> %dep
>> z.load("your.org:your-artifact:1.0.0")
>>
>> %spark
>> import your.org.your-artifact._
>> val app = new YourApp(sc)
>>
>> If it is not the case your app can be used, yes we need some patch for
>> binding already running remote interpreter. (currently remote interpreter
>> can be run manually, but zeppelin server has no option to bind to it)
>>
>> Thanks,
>> moon
>>
>>
>> On Mon, May 18, 2015 at 3:06 PM Harsh Rathi <co...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Yes.
>>> Since my app is part of system and I want to provide Zeppelin as an
>>> additional feature running Custom Interpreter. This custom interpreter will
>>> use my app's SparkContext.
>>>
>>> I looked more into the docs and came to know about Remote Interpreter.
>>> So, from my app if I run Zeppelin Server and a Remote Custom Interpreter
>>> having my Spark Context, then, attach this interpreter to Zeppelin Server.
>>> I guess that would be sufficient for me.
>>>
>>> I scanned through all docs, but couldn't find a demo of remote
>>> interpreter. If you can provide me the steps (or commands) to do it, that
>>> will be very helpful.
>>>
>>> On Mon, May 18, 2015 at 10:56 AM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> You mean share SparkContext between Zeppelin and your app, right?
>>>> If they're running in different JVM, i don't think there're good way of
>>>> share.
>>>>
>>>> Can i ask why do you want to share SparkContext?
>>>>
>>>> Best,
>>>> moon
>>>>
>>>> On Mon, May 18, 2015 at 2:00 PM Harsh Rathi <co...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Anything in this regard ?
>>>>>
>>>>>
>>>>> Harsh Rathi
>>>>>
>>>>>
>>>>> On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <coolharshrathi@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am new to Zeppelin.
>>>>>>
>>>>>> I want to share Spark Context with Zeppelin.
>>>>>> I want to invoke zeppelin and pass Spark Context to it so that
>>>>>> whenever my app is started, zeppelin gets started and my spark context is
>>>>>> used by Zeppelin. Is there any way to do it ?
>>>>>>
>>>>>> Harsh Rathi
>>>>>>
>>>>>>
>>>>>
>>>
>
Re: Invoking and passing SparkContext to Zeppelin
Posted by Harsh Rathi <co...@gmail.com>.
Hi,
I think I am getting confused here.
So, what is the purpose of remote interpreter ?
How remote server will be used if it can't connect to Zeppelin Server ?
Is there any way I can invoke and pass custom arguments at run time to my
Custom Interpreter which may run alongwith Zeppelin Server ?
Thanks
Harsh
On Tue, May 19, 2015 at 4:54 AM, moon soo Lee <mo...@apache.org> wrote:
> Hi,
>
> One completely difference possible approach is, you can run your app
> inside of Zeppelin. Something like
>
> %dep
> z.load("your.org:your-artifact:1.0.0")
>
> %spark
> import your.org.your-artifact._
> val app = new YourApp(sc)
>
> If it is not the case your app can be used, yes we need some patch for
> binding already running remote interpreter. (currently remote interpreter
> can be run manually, but zeppelin server has no option to bind to it)
>
> Thanks,
> moon
>
>
> On Mon, May 18, 2015 at 3:06 PM Harsh Rathi <co...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Yes.
>> Since my app is part of system and I want to provide Zeppelin as an
>> additional feature running Custom Interpreter. This custom interpreter will
>> use my app's SparkContext.
>>
>> I looked more into the docs and came to know about Remote Interpreter.
>> So, from my app if I run Zeppelin Server and a Remote Custom Interpreter
>> having my Spark Context, then, attach this interpreter to Zeppelin Server.
>> I guess that would be sufficient for me.
>>
>> I scanned through all docs, but couldn't find a demo of remote
>> interpreter. If you can provide me the steps (or commands) to do it, that
>> will be very helpful.
>>
>> On Mon, May 18, 2015 at 10:56 AM, moon soo Lee <mo...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> You mean share SparkContext between Zeppelin and your app, right?
>>> If they're running in different JVM, i don't think there're good way of
>>> share.
>>>
>>> Can i ask why do you want to share SparkContext?
>>>
>>> Best,
>>> moon
>>>
>>> On Mon, May 18, 2015 at 2:00 PM Harsh Rathi <co...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Anything in this regard ?
>>>>
>>>>
>>>> Harsh Rathi
>>>>
>>>>
>>>> On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <co...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am new to Zeppelin.
>>>>>
>>>>> I want to share Spark Context with Zeppelin.
>>>>> I want to invoke zeppelin and pass Spark Context to it so that
>>>>> whenever my app is started, zeppelin gets started and my spark context is
>>>>> used by Zeppelin. Is there any way to do it ?
>>>>>
>>>>> Harsh Rathi
>>>>>
>>>>>
>>>>
>>
Re: Invoking and passing SparkContext to Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
One completely difference possible approach is, you can run your app inside
of Zeppelin. Something like
%dep
z.load("your.org:your-artifact:1.0.0")
%spark
import your.org.your-artifact._
val app = new YourApp(sc)
If it is not the case your app can be used, yes we need some patch for
binding already running remote interpreter. (currently remote interpreter
can be run manually, but zeppelin server has no option to bind to it)
Thanks,
moon
On Mon, May 18, 2015 at 3:06 PM Harsh Rathi <co...@gmail.com>
wrote:
> Hi,
>
> Yes.
> Since my app is part of system and I want to provide Zeppelin as an
> additional feature running Custom Interpreter. This custom interpreter will
> use my app's SparkContext.
>
> I looked more into the docs and came to know about Remote Interpreter.
> So, from my app if I run Zeppelin Server and a Remote Custom Interpreter
> having my Spark Context, then, attach this interpreter to Zeppelin Server.
> I guess that would be sufficient for me.
>
> I scanned through all docs, but couldn't find a demo of remote
> interpreter. If you can provide me the steps (or commands) to do it, that
> will be very helpful.
>
> On Mon, May 18, 2015 at 10:56 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi,
>>
>> You mean share SparkContext between Zeppelin and your app, right?
>> If they're running in different JVM, i don't think there're good way of
>> share.
>>
>> Can i ask why do you want to share SparkContext?
>>
>> Best,
>> moon
>>
>> On Mon, May 18, 2015 at 2:00 PM Harsh Rathi <co...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Anything in this regard ?
>>>
>>>
>>> Harsh Rathi
>>>
>>>
>>> On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <co...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to Zeppelin.
>>>>
>>>> I want to share Spark Context with Zeppelin.
>>>> I want to invoke zeppelin and pass Spark Context to it so that whenever
>>>> my app is started, zeppelin gets started and my spark context is used by
>>>> Zeppelin. Is there any way to do it ?
>>>>
>>>> Harsh Rathi
>>>>
>>>>
>>>
>
Re: Invoking and passing SparkContext to Zeppelin
Posted by Harsh Rathi <co...@gmail.com>.
Hi,
Yes.
Since my app is part of system and I want to provide Zeppelin as an
additional feature running Custom Interpreter. This custom interpreter will
use my app's SparkContext.
I looked more into the docs and came to know about Remote Interpreter.
So, from my app if I run Zeppelin Server and a Remote Custom Interpreter
having my Spark Context, then, attach this interpreter to Zeppelin Server.
I guess that would be sufficient for me.
I scanned through all docs, but couldn't find a demo of remote interpreter.
If you can provide me the steps (or commands) to do it, that will be very
helpful.
On Mon, May 18, 2015 at 10:56 AM, moon soo Lee <mo...@apache.org> wrote:
> Hi,
>
> You mean share SparkContext between Zeppelin and your app, right?
> If they're running in different JVM, i don't think there're good way of
> share.
>
> Can i ask why do you want to share SparkContext?
>
> Best,
> moon
>
> On Mon, May 18, 2015 at 2:00 PM Harsh Rathi <co...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Anything in this regard ?
>>
>>
>> Harsh Rathi
>>
>>
>> On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <co...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am new to Zeppelin.
>>>
>>> I want to share Spark Context with Zeppelin.
>>> I want to invoke zeppelin and pass Spark Context to it so that whenever
>>> my app is started, zeppelin gets started and my spark context is used by
>>> Zeppelin. Is there any way to do it ?
>>>
>>> Harsh Rathi
>>>
>>>
>>
Re: Invoking and passing SparkContext to Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
You mean share SparkContext between Zeppelin and your app, right?
If they're running in different JVM, i don't think there're good way of
share.
Can i ask why do you want to share SparkContext?
Best,
moon
On Mon, May 18, 2015 at 2:00 PM Harsh Rathi <co...@gmail.com>
wrote:
> Hi,
>
> Anything in this regard ?
>
>
> Harsh Rathi
>
>
> On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <co...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am new to Zeppelin.
>>
>> I want to share Spark Context with Zeppelin.
>> I want to invoke zeppelin and pass Spark Context to it so that whenever
>> my app is started, zeppelin gets started and my spark context is used by
>> Zeppelin. Is there any way to do it ?
>>
>> Harsh Rathi
>>
>>
>
Re: Invoking and passing SparkContext to Zeppelin
Posted by Harsh Rathi <co...@gmail.com>.
Hi,
Anything in this regard ?
Harsh Rathi
On Fri, May 15, 2015 at 6:32 PM, Harsh Rathi <co...@gmail.com>
wrote:
> Hi,
>
> I am new to Zeppelin.
>
> I want to share Spark Context with Zeppelin.
> I want to invoke zeppelin and pass Spark Context to it so that whenever my
> app is started, zeppelin gets started and my spark context is used by
> Zeppelin. Is there any way to do it ?
>
> Harsh Rathi
>
>