You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by IT CTO <go...@gmail.com> on 2015/07/12 08:10:37 UTC
ZEPPELIN-163 zeppelin interpreter loading process
please help me understand the interpreter configuration process
1) When zeppelin first start it looks for interpreter.json in the conf dir
2) if found, uses that json for the configuration shown in the config menu.
3) if not found, ??? traverse the interpreter directory for all available
interpreters? where does it get the default parameters from?
4) when user click Create+ button on the interpreter menu, the form shows a
name and the interpreter combo-box is derived from the interpreter REST API
that in turn gets its value from the interpreterFactory(
https://github.com/apache/incubator-zeppelin/blob/12e5abf2803e4c5015998672b10642fc72aac0da/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java#L94)
.
*if no zeppelin-site.xml is present, where does the conf is taken from?*
5) current docs say we should add the new interpreter to the site.xml file
- but all other interpreters are not in this file (actually only a template
file is given with a fresh install) - I think docs should say something
about "defaults interpreters which "ship" with zeppelin and other
interpreters - or on the other hand this step is not needed?
Thanks for the opportunity to take part in this project :-)
Eran
On Thu, Jul 9, 2015 at 6:36 PM Eran W (JIRA) <ji...@apache.org> wrote:
> Eran W created ZEPPELIN-163:
> -------------------------------
>
> Summary: Change Writing Zeppelin Interpreter documentation
> page
> Key: ZEPPELIN-163
> URL: https://issues.apache.org/jira/browse/ZEPPELIN-163
> Project: Zeppelin
> Issue Type: Improvement
> Reporter: Eran W
> Priority: Minor
>
>
> 1) change wording on What is Zeppelin Interpreter
> Interpreter can be launched either using separate JVM process or separate
> classloader. The defualt is using separate JVM process (by checking 'fork'
> in Interpreter menu). When Interpreter is running in separate JVM process,
> it's communicating with Zeppelin via thrift. The other option is to use
> separate classloader, note that somtime separate classloader can causes
> problem especially when your interpreter uses reflections or trying to grab
> standard out/err.
>
> 2) Add to Configure your interpreter
> What do we need to do for the interpreter to show up in the interpreter
> menu...
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
Re: ZEPPELIN-163 zeppelin interpreter loading process
Posted by moon soo Lee <mo...@apache.org>.
About question
*if no zeppelin-site.xml is present, where does the conf is taken from?*
Default value of configurations are defined
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java#L397
Thanks,
moon
On Sun, Jul 12, 2015 at 10:22 PM IT CTO <go...@gmail.com> wrote:
> Hi,
> Thanks for the detailed explanation I followed the code path and got to the
> same results. I am going to update the documentation as right now the
> process for an "internal\community interpreter" is different from the
> process for a private interpreter (e.g. one that is not part of the public
> build) also, I think that one should not need to follow the code as you did
> to figure this out.
> Thanks.
> Eran
>
> On Sun, Jul 12, 2015 at 10:46 PM DuyHai Doan <do...@gmail.com> wrote:
>
> > Hello Eran
> >
> > I'm developing right now a Cassandra Interpreter so I ran into the same
> > questions.
> >
> > 3) if not found, ??? traverse the interpreter directory for all available
> > interpreters? where does it get the default parameters from?
> >
> > All the bootstrap process is inside
> > org.apache.zeppelin.interpreter.InterpreterFactory
> >
> > a. when the factory is created, it first accesses the class
> > ZeppelinConfiguration and look for the attribute "ZEPPELIN_INTERPRETERS"
> to
> > get a list of all possible interpreter classes
> >
> > b. upon call on method init(), Zeppelin will scan the folder "
> > zeppelin.interpreter.dir" (default value = $ZEPPELIN_HOME/interpreter).
> For
> > each found interpreter, it will look at the interpreter classes found
> > previously at step a.
> >
> > c. then the ZeppelinFactory will call Class.forName("interpreter fully
> > qualified class name", true) to initialise the static block in your
> > Interpreter class. Normally your interpreter class should have the
> > following static block:
> >
> > static {
> > Interpreter.register("dummyInterpreter", "dummyInterpreter",
> > MyInterpreter.class.getName(),
> > new InterpreterPropertyBuilder()
> > .add(FIRST_PROPERTY, DEFAULT_VALUE,"Description of this property")
> > ...
> > );
> > }
> >
> >
> > When you click on the "Interpreter" menu on the web interface to create a
> > new interpreter setting, the default conf is taken from this static init
> > block
> >
> > Hope that helps
> >
> >
> >
> >
> > On Sun, Jul 12, 2015 at 8:10 AM, IT CTO <go...@gmail.com> wrote:
> >
> > > please help me understand the interpreter configuration process
> > >
> > > 1) When zeppelin first start it looks for interpreter.json in the conf
> > dir
> > > 2) if found, uses that json for the configuration shown in the config
> > menu.
> > > 3) if not found, ??? traverse the interpreter directory for all
> available
> > > interpreters? where does it get the default parameters from?
> > > 4) when user click Create+ button on the interpreter menu, the form
> > shows a
> > > name and the interpreter combo-box is derived from the interpreter REST
> > API
> > > that in turn gets its value from the interpreterFactory(
> > >
> > >
> >
> https://github.com/apache/incubator-zeppelin/blob/12e5abf2803e4c5015998672b10642fc72aac0da/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java#L94
> > > )
> > > .
> > > *if no zeppelin-site.xml is present, where does the conf is taken
> from?*
> > >
> > > 5) current docs say we should add the new interpreter to the site.xml
> > file
> > > - but all other interpreters are not in this file (actually only a
> > template
> > > file is given with a fresh install) - I think docs should say something
> > > about "defaults interpreters which "ship" with zeppelin and other
> > > interpreters - or on the other hand this step is not needed?
> > >
> > > Thanks for the opportunity to take part in this project :-)
> > > Eran
> > >
> > > On Thu, Jul 9, 2015 at 6:36 PM Eran W (JIRA) <ji...@apache.org> wrote:
> > >
> > > > Eran W created ZEPPELIN-163:
> > > > -------------------------------
> > > >
> > > > Summary: Change Writing Zeppelin Interpreter
> documentation
> > > > page
> > > > Key: ZEPPELIN-163
> > > > URL:
> > https://issues.apache.org/jira/browse/ZEPPELIN-163
> > > > Project: Zeppelin
> > > > Issue Type: Improvement
> > > > Reporter: Eran W
> > > > Priority: Minor
> > > >
> > > >
> > > > 1) change wording on What is Zeppelin Interpreter
> > > > Interpreter can be launched either using separate JVM process or
> > separate
> > > > classloader. The defualt is using separate JVM process (by checking
> > > 'fork'
> > > > in Interpreter menu). When Interpreter is running in separate JVM
> > > process,
> > > > it's communicating with Zeppelin via thrift. The other option is to
> use
> > > > separate classloader, note that somtime separate classloader can
> causes
> > > > problem especially when your interpreter uses reflections or trying
> to
> > > grab
> > > > standard out/err.
> > > >
> > > > 2) Add to Configure your interpreter
> > > > What do we need to do for the interpreter to show up in the
> interpreter
> > > > menu...
> > > >
> > > >
> > > >
> > > > --
> > > > This message was sent by Atlassian JIRA
> > > > (v6.3.4#6332)
> > > >
> > >
> >
>
Re: ZEPPELIN-163 zeppelin interpreter loading process
Posted by IT CTO <go...@gmail.com>.
Hi,
Thanks for the detailed explanation I followed the code path and got to the
same results. I am going to update the documentation as right now the
process for an "internal\community interpreter" is different from the
process for a private interpreter (e.g. one that is not part of the public
build) also, I think that one should not need to follow the code as you did
to figure this out.
Thanks.
Eran
On Sun, Jul 12, 2015 at 10:46 PM DuyHai Doan <do...@gmail.com> wrote:
> Hello Eran
>
> I'm developing right now a Cassandra Interpreter so I ran into the same
> questions.
>
> 3) if not found, ??? traverse the interpreter directory for all available
> interpreters? where does it get the default parameters from?
>
> All the bootstrap process is inside
> org.apache.zeppelin.interpreter.InterpreterFactory
>
> a. when the factory is created, it first accesses the class
> ZeppelinConfiguration and look for the attribute "ZEPPELIN_INTERPRETERS" to
> get a list of all possible interpreter classes
>
> b. upon call on method init(), Zeppelin will scan the folder "
> zeppelin.interpreter.dir" (default value = $ZEPPELIN_HOME/interpreter). For
> each found interpreter, it will look at the interpreter classes found
> previously at step a.
>
> c. then the ZeppelinFactory will call Class.forName("interpreter fully
> qualified class name", true) to initialise the static block in your
> Interpreter class. Normally your interpreter class should have the
> following static block:
>
> static {
> Interpreter.register("dummyInterpreter", "dummyInterpreter",
> MyInterpreter.class.getName(),
> new InterpreterPropertyBuilder()
> .add(FIRST_PROPERTY, DEFAULT_VALUE,"Description of this property")
> ...
> );
> }
>
>
> When you click on the "Interpreter" menu on the web interface to create a
> new interpreter setting, the default conf is taken from this static init
> block
>
> Hope that helps
>
>
>
>
> On Sun, Jul 12, 2015 at 8:10 AM, IT CTO <go...@gmail.com> wrote:
>
> > please help me understand the interpreter configuration process
> >
> > 1) When zeppelin first start it looks for interpreter.json in the conf
> dir
> > 2) if found, uses that json for the configuration shown in the config
> menu.
> > 3) if not found, ??? traverse the interpreter directory for all available
> > interpreters? where does it get the default parameters from?
> > 4) when user click Create+ button on the interpreter menu, the form
> shows a
> > name and the interpreter combo-box is derived from the interpreter REST
> API
> > that in turn gets its value from the interpreterFactory(
> >
> >
> https://github.com/apache/incubator-zeppelin/blob/12e5abf2803e4c5015998672b10642fc72aac0da/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java#L94
> > )
> > .
> > *if no zeppelin-site.xml is present, where does the conf is taken from?*
> >
> > 5) current docs say we should add the new interpreter to the site.xml
> file
> > - but all other interpreters are not in this file (actually only a
> template
> > file is given with a fresh install) - I think docs should say something
> > about "defaults interpreters which "ship" with zeppelin and other
> > interpreters - or on the other hand this step is not needed?
> >
> > Thanks for the opportunity to take part in this project :-)
> > Eran
> >
> > On Thu, Jul 9, 2015 at 6:36 PM Eran W (JIRA) <ji...@apache.org> wrote:
> >
> > > Eran W created ZEPPELIN-163:
> > > -------------------------------
> > >
> > > Summary: Change Writing Zeppelin Interpreter documentation
> > > page
> > > Key: ZEPPELIN-163
> > > URL:
> https://issues.apache.org/jira/browse/ZEPPELIN-163
> > > Project: Zeppelin
> > > Issue Type: Improvement
> > > Reporter: Eran W
> > > Priority: Minor
> > >
> > >
> > > 1) change wording on What is Zeppelin Interpreter
> > > Interpreter can be launched either using separate JVM process or
> separate
> > > classloader. The defualt is using separate JVM process (by checking
> > 'fork'
> > > in Interpreter menu). When Interpreter is running in separate JVM
> > process,
> > > it's communicating with Zeppelin via thrift. The other option is to use
> > > separate classloader, note that somtime separate classloader can causes
> > > problem especially when your interpreter uses reflections or trying to
> > grab
> > > standard out/err.
> > >
> > > 2) Add to Configure your interpreter
> > > What do we need to do for the interpreter to show up in the interpreter
> > > menu...
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian JIRA
> > > (v6.3.4#6332)
> > >
> >
>
Re: ZEPPELIN-163 zeppelin interpreter loading process
Posted by DuyHai Doan <do...@gmail.com>.
Hello Eran
I'm developing right now a Cassandra Interpreter so I ran into the same
questions.
3) if not found, ??? traverse the interpreter directory for all available
interpreters? where does it get the default parameters from?
All the bootstrap process is inside
org.apache.zeppelin.interpreter.InterpreterFactory
a. when the factory is created, it first accesses the class
ZeppelinConfiguration and look for the attribute "ZEPPELIN_INTERPRETERS" to
get a list of all possible interpreter classes
b. upon call on method init(), Zeppelin will scan the folder "
zeppelin.interpreter.dir" (default value = $ZEPPELIN_HOME/interpreter). For
each found interpreter, it will look at the interpreter classes found
previously at step a.
c. then the ZeppelinFactory will call Class.forName("interpreter fully
qualified class name", true) to initialise the static block in your
Interpreter class. Normally your interpreter class should have the
following static block:
static {
Interpreter.register("dummyInterpreter", "dummyInterpreter",
MyInterpreter.class.getName(),
new InterpreterPropertyBuilder()
.add(FIRST_PROPERTY, DEFAULT_VALUE,"Description of this property")
...
);
}
When you click on the "Interpreter" menu on the web interface to create a
new interpreter setting, the default conf is taken from this static init
block
Hope that helps
On Sun, Jul 12, 2015 at 8:10 AM, IT CTO <go...@gmail.com> wrote:
> please help me understand the interpreter configuration process
>
> 1) When zeppelin first start it looks for interpreter.json in the conf dir
> 2) if found, uses that json for the configuration shown in the config menu.
> 3) if not found, ??? traverse the interpreter directory for all available
> interpreters? where does it get the default parameters from?
> 4) when user click Create+ button on the interpreter menu, the form shows a
> name and the interpreter combo-box is derived from the interpreter REST API
> that in turn gets its value from the interpreterFactory(
>
> https://github.com/apache/incubator-zeppelin/blob/12e5abf2803e4c5015998672b10642fc72aac0da/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java#L94
> )
> .
> *if no zeppelin-site.xml is present, where does the conf is taken from?*
>
> 5) current docs say we should add the new interpreter to the site.xml file
> - but all other interpreters are not in this file (actually only a template
> file is given with a fresh install) - I think docs should say something
> about "defaults interpreters which "ship" with zeppelin and other
> interpreters - or on the other hand this step is not needed?
>
> Thanks for the opportunity to take part in this project :-)
> Eran
>
> On Thu, Jul 9, 2015 at 6:36 PM Eran W (JIRA) <ji...@apache.org> wrote:
>
> > Eran W created ZEPPELIN-163:
> > -------------------------------
> >
> > Summary: Change Writing Zeppelin Interpreter documentation
> > page
> > Key: ZEPPELIN-163
> > URL: https://issues.apache.org/jira/browse/ZEPPELIN-163
> > Project: Zeppelin
> > Issue Type: Improvement
> > Reporter: Eran W
> > Priority: Minor
> >
> >
> > 1) change wording on What is Zeppelin Interpreter
> > Interpreter can be launched either using separate JVM process or separate
> > classloader. The defualt is using separate JVM process (by checking
> 'fork'
> > in Interpreter menu). When Interpreter is running in separate JVM
> process,
> > it's communicating with Zeppelin via thrift. The other option is to use
> > separate classloader, note that somtime separate classloader can causes
> > problem especially when your interpreter uses reflections or trying to
> grab
> > standard out/err.
> >
> > 2) Add to Configure your interpreter
> > What do we need to do for the interpreter to show up in the interpreter
> > menu...
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>