You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Jongyoul Lee <jo...@gmail.com> on 2016/10/04 07:00:52 UTC
How Apache Zeppelin runs a paragraph
Hello devs and users,
I've submitted a post into Apache Zeppelin Stories on medium.com:
https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
I hope all users and contributors understand Apache Zeppelin deeply.
Regards,
Jongyoul
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Jongyoul Lee <jo...@gmail.com>.
Thanks you Jeff for the comments,
1. You're right, so some contribution already is submitted and merged.
Tagging these kind of bugs as `newbie` and `beginner` would be helpful.
2. I'll plan to write a post about Interpreters including this topic.
Thanks,
Jongyoul
On Sun, Oct 9, 2016 at 9:29 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Thanks Jougyoul for the effort. This is really helpful for new
> contributors.
>
> Several comments.
>
> 1. Seems we didn't use noteId/notebookId correctly in code, do we have
> ticket to track that ? Because the sample code is conflicted with the
> definition of note/notebook in the post, it might confuse users.
>
> @Path("job/{notebookId}/{paragraphId}")
> @ZeppelinApi
> public Response runParagraph(@PathParam("notebookId") String notebookId,
> @PathParam("paragraphId") String paragraphId, String message)
> throws IOException, IllegalArgumentException {
> LOG.info("run paragraph job asynchronously {} {} {}", notebookId, paragraphId, message);
>
> Note note = notebook.getNote(notebookId);
>
>
> 2. It might be better to use diagram to visualize the differences between shared/scoped/shared
>
>
>
>
> On Sun, Oct 9, 2016 at 12:38 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> Great post!
>>
>>
>>
>>
>>
>> On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>
>> wrote:
>>
>> Hello DuyHai,
>>
>> Thanks for the fixing the typo. I've fixed it.
>>
>> Concerning the debugging, I think writing posts or updating wiki would be
>> better. I'm willing to write a simple post for that remote debugging.
>>
>> On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
>>
>>> Hello Jongyoul
>>>
>>> Excellent write up to demystify the execution life-cycle of a paragraph.
>>>
>>> There is a small typo in the blog: "At last, it has two words separated
>>> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_na
>>> me}”"
>>>
>>> Instead of comma, shouldn't it be DOT (.) ?
>>>
>>> There is also another unknown details that may be interesting to expose:
>>> how are interpreter JVM launched ?
>>>
>>> InterpreterFactory
>>>
>>> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
>>> Properties property, String interpreterSettingId) {
>>> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
>>> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
>>> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>>>
>>> RemoteInterpreter remoteInterpreter =
>>> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
>>> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
>>> remoteInterpreterProcessListener, appEventListener);
>>> remoteInterpreter.addEnv(env);
>>>
>>> return new LazyOpenInterpreter(remoteInterpreter);
>>> }
>>>
>>>
>>> The code will call the bin/intepreter.sh (or interpreter.cmd if on
>>> Windows) to launch a new JVM for the interpreter.
>>>
>>> And depending on the type of interpreter, the command will be different:
>>>
>>> if [[ -n "${SPARK_SUBMIT}" ]]; then
>>> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
>>> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}"
>>> --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS}
>>> ${SPARK_APP_JAR} ${PORT} &
>>> else
>>> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
>>> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
>>> ${PORT} &
>>> fi
>>>
>>> There is a special case for Spark because we execute a SparkSubmit.
>>>
>>> This detail is important for contributors because when they need to
>>> DEBUG an interpreter code, they have to modify the interpreter.sh file to
>>> add an "-Xdebug -Xrunjdwp:transport=dt_socket,
>>> server=y,suspend=n,address=xxx"
>>>
>>> Most of people naively add this debug argument to the zeppelin.sh file
>>> but it only debugs the Zeppelin server itself, not the interpreter.
>>>
>>> Do you think we may create a page in the official doc for this ?
>>>
>>>
>>> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>>
>>>> Hello devs and users,
>>>>
>>>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>>>> https://medium.com/apache-zeppelin-stories/how-a
>>>> pache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>>>
>>>> I hope all users and contributors understand Apache Zeppelin deeply.
>>>>
>>>> Regards,
>>>> Jongyoul
>>>>
>>>> --
>>>> 이종열, Jongyoul Lee, 李宗烈
>>>> http://madeng.net
>>>>
>>>
>>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Jongyoul Lee <jo...@gmail.com>.
Thanks you Jeff for the comments,
1. You're right, so some contribution already is submitted and merged.
Tagging these kind of bugs as `newbie` and `beginner` would be helpful.
2. I'll plan to write a post about Interpreters including this topic.
Thanks,
Jongyoul
On Sun, Oct 9, 2016 at 9:29 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Thanks Jougyoul for the effort. This is really helpful for new
> contributors.
>
> Several comments.
>
> 1. Seems we didn't use noteId/notebookId correctly in code, do we have
> ticket to track that ? Because the sample code is conflicted with the
> definition of note/notebook in the post, it might confuse users.
>
> @Path("job/{notebookId}/{paragraphId}")
> @ZeppelinApi
> public Response runParagraph(@PathParam("notebookId") String notebookId,
> @PathParam("paragraphId") String paragraphId, String message)
> throws IOException, IllegalArgumentException {
> LOG.info("run paragraph job asynchronously {} {} {}", notebookId, paragraphId, message);
>
> Note note = notebook.getNote(notebookId);
>
>
> 2. It might be better to use diagram to visualize the differences between shared/scoped/shared
>
>
>
>
> On Sun, Oct 9, 2016 at 12:38 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> Great post!
>>
>>
>>
>>
>>
>> On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>
>> wrote:
>>
>> Hello DuyHai,
>>
>> Thanks for the fixing the typo. I've fixed it.
>>
>> Concerning the debugging, I think writing posts or updating wiki would be
>> better. I'm willing to write a simple post for that remote debugging.
>>
>> On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
>>
>>> Hello Jongyoul
>>>
>>> Excellent write up to demystify the execution life-cycle of a paragraph.
>>>
>>> There is a small typo in the blog: "At last, it has two words separated
>>> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_na
>>> me}”"
>>>
>>> Instead of comma, shouldn't it be DOT (.) ?
>>>
>>> There is also another unknown details that may be interesting to expose:
>>> how are interpreter JVM launched ?
>>>
>>> InterpreterFactory
>>>
>>> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
>>> Properties property, String interpreterSettingId) {
>>> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
>>> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
>>> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>>>
>>> RemoteInterpreter remoteInterpreter =
>>> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
>>> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
>>> remoteInterpreterProcessListener, appEventListener);
>>> remoteInterpreter.addEnv(env);
>>>
>>> return new LazyOpenInterpreter(remoteInterpreter);
>>> }
>>>
>>>
>>> The code will call the bin/intepreter.sh (or interpreter.cmd if on
>>> Windows) to launch a new JVM for the interpreter.
>>>
>>> And depending on the type of interpreter, the command will be different:
>>>
>>> if [[ -n "${SPARK_SUBMIT}" ]]; then
>>> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
>>> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}"
>>> --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS}
>>> ${SPARK_APP_JAR} ${PORT} &
>>> else
>>> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
>>> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
>>> ${PORT} &
>>> fi
>>>
>>> There is a special case for Spark because we execute a SparkSubmit.
>>>
>>> This detail is important for contributors because when they need to
>>> DEBUG an interpreter code, they have to modify the interpreter.sh file to
>>> add an "-Xdebug -Xrunjdwp:transport=dt_socket,
>>> server=y,suspend=n,address=xxx"
>>>
>>> Most of people naively add this debug argument to the zeppelin.sh file
>>> but it only debugs the Zeppelin server itself, not the interpreter.
>>>
>>> Do you think we may create a page in the official doc for this ?
>>>
>>>
>>> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>>
>>>> Hello devs and users,
>>>>
>>>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>>>> https://medium.com/apache-zeppelin-stories/how-a
>>>> pache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>>>
>>>> I hope all users and contributors understand Apache Zeppelin deeply.
>>>>
>>>> Regards,
>>>> Jongyoul
>>>>
>>>> --
>>>> 이종열, Jongyoul Lee, 李宗烈
>>>> http://madeng.net
>>>>
>>>
>>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Jeff Zhang <zj...@gmail.com>.
Thanks Jougyoul for the effort. This is really helpful for new contributors.
Several comments.
1. Seems we didn't use noteId/notebookId correctly in code, do we have
ticket to track that ? Because the sample code is conflicted with the
definition of note/notebook in the post, it might confuse users.
@Path("job/{notebookId}/{paragraphId}")
@ZeppelinApi
public Response runParagraph(@PathParam("notebookId") String notebookId,
@PathParam("paragraphId") String paragraphId, String message)
throws IOException, IllegalArgumentException {
LOG.info("run paragraph job asynchronously {} {} {}", notebookId,
paragraphId, message);
Note note = notebook.getNote(notebookId);
2. It might be better to use diagram to visualize the differences
between shared/scoped/shared
On Sun, Oct 9, 2016 at 12:38 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> Great post!
>
>
>
>
>
> On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>
> wrote:
>
> Hello DuyHai,
>
> Thanks for the fixing the typo. I've fixed it.
>
> Concerning the debugging, I think writing posts or updating wiki would be
> better. I'm willing to write a simple post for that remote debugging.
>
> On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
>
>> Hello Jongyoul
>>
>> Excellent write up to demystify the execution life-cycle of a paragraph.
>>
>> There is a small typo in the blog: "At last, it has two words separated
>> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_na
>> me}”"
>>
>> Instead of comma, shouldn't it be DOT (.) ?
>>
>> There is also another unknown details that may be interesting to expose:
>> how are interpreter JVM launched ?
>>
>> InterpreterFactory
>>
>> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
>> Properties property, String interpreterSettingId) {
>> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
>> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
>> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>>
>> RemoteInterpreter remoteInterpreter =
>> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
>> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
>> remoteInterpreterProcessListener, appEventListener);
>> remoteInterpreter.addEnv(env);
>>
>> return new LazyOpenInterpreter(remoteInterpreter);
>> }
>>
>>
>> The code will call the bin/intepreter.sh (or interpreter.cmd if on
>> Windows) to launch a new JVM for the interpreter.
>>
>> And depending on the type of interpreter, the command will be different:
>>
>> if [[ -n "${SPARK_SUBMIT}" ]]; then
>> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
>> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}"
>> --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS}
>> ${SPARK_APP_JAR} ${PORT} &
>> else
>> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
>> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
>> ${PORT} &
>> fi
>>
>> There is a special case for Spark because we execute a SparkSubmit.
>>
>> This detail is important for contributors because when they need to DEBUG
>> an interpreter code, they have to modify the interpreter.sh file to add an
>> "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
>>
>> Most of people naively add this debug argument to the zeppelin.sh file
>> but it only debugs the Zeppelin server itself, not the interpreter.
>>
>> Do you think we may create a page in the official doc for this ?
>>
>>
>> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>
>>> Hello devs and users,
>>>
>>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>>> https://medium.com/apache-zeppelin-stories/how-a
>>> pache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>>
>>> I hope all users and contributors understand Apache Zeppelin deeply.
>>>
>>> Regards,
>>> Jongyoul
>>>
>>> --
>>> 이종열, Jongyoul Lee, 李宗烈
>>> http://madeng.net
>>>
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
--
Best Regards
Jeff Zhang
Re: How Apache Zeppelin runs a paragraph
Posted by Jeff Zhang <zj...@gmail.com>.
Thanks Jougyoul for the effort. This is really helpful for new contributors.
Several comments.
1. Seems we didn't use noteId/notebookId correctly in code, do we have
ticket to track that ? Because the sample code is conflicted with the
definition of note/notebook in the post, it might confuse users.
@Path("job/{notebookId}/{paragraphId}")
@ZeppelinApi
public Response runParagraph(@PathParam("notebookId") String notebookId,
@PathParam("paragraphId") String paragraphId, String message)
throws IOException, IllegalArgumentException {
LOG.info("run paragraph job asynchronously {} {} {}", notebookId,
paragraphId, message);
Note note = notebook.getNote(notebookId);
2. It might be better to use diagram to visualize the differences
between shared/scoped/shared
On Sun, Oct 9, 2016 at 12:38 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> Great post!
>
>
>
>
>
> On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>
> wrote:
>
> Hello DuyHai,
>
> Thanks for the fixing the typo. I've fixed it.
>
> Concerning the debugging, I think writing posts or updating wiki would be
> better. I'm willing to write a simple post for that remote debugging.
>
> On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
>
>> Hello Jongyoul
>>
>> Excellent write up to demystify the execution life-cycle of a paragraph.
>>
>> There is a small typo in the blog: "At last, it has two words separated
>> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_na
>> me}”"
>>
>> Instead of comma, shouldn't it be DOT (.) ?
>>
>> There is also another unknown details that may be interesting to expose:
>> how are interpreter JVM launched ?
>>
>> InterpreterFactory
>>
>> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
>> Properties property, String interpreterSettingId) {
>> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
>> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
>> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>>
>> RemoteInterpreter remoteInterpreter =
>> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
>> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
>> remoteInterpreterProcessListener, appEventListener);
>> remoteInterpreter.addEnv(env);
>>
>> return new LazyOpenInterpreter(remoteInterpreter);
>> }
>>
>>
>> The code will call the bin/intepreter.sh (or interpreter.cmd if on
>> Windows) to launch a new JVM for the interpreter.
>>
>> And depending on the type of interpreter, the command will be different:
>>
>> if [[ -n "${SPARK_SUBMIT}" ]]; then
>> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
>> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}"
>> --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS}
>> ${SPARK_APP_JAR} ${PORT} &
>> else
>> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
>> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
>> ${PORT} &
>> fi
>>
>> There is a special case for Spark because we execute a SparkSubmit.
>>
>> This detail is important for contributors because when they need to DEBUG
>> an interpreter code, they have to modify the interpreter.sh file to add an
>> "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
>>
>> Most of people naively add this debug argument to the zeppelin.sh file
>> but it only debugs the Zeppelin server itself, not the interpreter.
>>
>> Do you think we may create a page in the official doc for this ?
>>
>>
>> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>
>>> Hello devs and users,
>>>
>>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>>> https://medium.com/apache-zeppelin-stories/how-a
>>> pache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>>
>>> I hope all users and contributors understand Apache Zeppelin deeply.
>>>
>>> Regards,
>>> Jongyoul
>>>
>>> --
>>> 이종열, Jongyoul Lee, 李宗烈
>>> http://madeng.net
>>>
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
--
Best Regards
Jeff Zhang
Re: How Apache Zeppelin runs a paragraph
Posted by Felix Cheung <fe...@hotmail.com>.
Great post!
On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>> wrote:
Hello DuyHai,
Thanks for the fixing the typo. I've fixed it.
Concerning the debugging, I think writing posts or updating wiki would be better. I'm willing to write a simple post for that remote debugging.
On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com>> wrote:
Hello Jongyoul
Excellent write up to demystify the execution life-cycle of a paragraph.
There is a small typo in the blog: "At last, it has two words separated by comma, getInterpreter handles it as "%{group_name}.{interpreter_name}""
Instead of comma, shouldn't it be DOT (.) ?
There is also another unknown details that may be interesting to expose: how are interpreter JVM launched ?
InterpreterFactory
private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
Properties property, String interpreterSettingId) {
int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
RemoteInterpreter remoteInterpreter =
new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
The code will call the bin/intepreter.sh (or interpreter.cmd if on Windows) to launch a new JVM for the interpreter.
And depending on the type of interpreter, the command will be different:
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
fi
There is a special case for Spark because we execute a SparkSubmit.
This detail is important for contributors because when they need to DEBUG an interpreter code, they have to modify the interpreter.sh file to add an "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
Most of people naively add this debug argument to the zeppelin.sh file but it only debugs the Zeppelin server itself, not the interpreter.
Do you think we may create a page in the official doc for this ?
On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com>> wrote:
Hello devs and users,
I've submitted a post into Apache Zeppelin Stories on medium.com<http://medium.com>: https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
I hope all users and contributors understand Apache Zeppelin deeply.
Regards,
Jongyoul
--
???, Jongyoul Lee, ???
http://madeng.net
--
???, Jongyoul Lee, ???
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Felix Cheung <fe...@hotmail.com>.
Great post!
On Tue, Oct 4, 2016 at 8:56 PM -0700, "Jongyoul Lee" <jo...@gmail.com>> wrote:
Hello DuyHai,
Thanks for the fixing the typo. I've fixed it.
Concerning the debugging, I think writing posts or updating wiki would be better. I'm willing to write a simple post for that remote debugging.
On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com>> wrote:
Hello Jongyoul
Excellent write up to demystify the execution life-cycle of a paragraph.
There is a small typo in the blog: "At last, it has two words separated by comma, getInterpreter handles it as "%{group_name}.{interpreter_name}""
Instead of comma, shouldn't it be DOT (.) ?
There is also another unknown details that may be interesting to expose: how are interpreter JVM launched ?
InterpreterFactory
private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
Properties property, String interpreterSettingId) {
int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
RemoteInterpreter remoteInterpreter =
new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
The code will call the bin/intepreter.sh (or interpreter.cmd if on Windows) to launch a new JVM for the interpreter.
And depending on the type of interpreter, the command will be different:
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
fi
There is a special case for Spark because we execute a SparkSubmit.
This detail is important for contributors because when they need to DEBUG an interpreter code, they have to modify the interpreter.sh file to add an "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
Most of people naively add this debug argument to the zeppelin.sh file but it only debugs the Zeppelin server itself, not the interpreter.
Do you think we may create a page in the official doc for this ?
On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com>> wrote:
Hello devs and users,
I've submitted a post into Apache Zeppelin Stories on medium.com<http://medium.com>: https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
I hope all users and contributors understand Apache Zeppelin deeply.
Regards,
Jongyoul
--
???, Jongyoul Lee, ???
http://madeng.net
--
???, Jongyoul Lee, ???
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Jongyoul Lee <jo...@gmail.com>.
Hello DuyHai,
Thanks for the fixing the typo. I've fixed it.
Concerning the debugging, I think writing posts or updating wiki would be
better. I'm willing to write a simple post for that remote debugging.
On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
> Hello Jongyoul
>
> Excellent write up to demystify the execution life-cycle of a paragraph.
>
> There is a small typo in the blog: "At last, it has two words separated
> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_
> name}”"
>
> Instead of comma, shouldn't it be DOT (.) ?
>
> There is also another unknown details that may be interesting to expose:
> how are interpreter JVM launched ?
>
> InterpreterFactory
>
> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
> Properties property, String interpreterSettingId) {
> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>
> RemoteInterpreter remoteInterpreter =
> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
> remoteInterpreterProcessListener, appEventListener);
> remoteInterpreter.addEnv(env);
>
> return new LazyOpenInterpreter(remoteInterpreter);
> }
>
>
> The code will call the bin/intepreter.sh (or interpreter.cmd if on
> Windows) to launch a new JVM for the interpreter.
>
> And depending on the type of interpreter, the command will be different:
>
> if [[ -n "${SPARK_SUBMIT}" ]]; then
> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options
> "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
> else
> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
> ${PORT} &
> fi
>
> There is a special case for Spark because we execute a SparkSubmit.
>
> This detail is important for contributors because when they need to DEBUG
> an interpreter code, they have to modify the interpreter.sh file to add an
> "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
>
> Most of people naively add this debug argument to the zeppelin.sh file but
> it only debugs the Zeppelin server itself, not the interpreter.
>
> Do you think we may create a page in the official doc for this ?
>
>
> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
>> Hello devs and users,
>>
>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>> https://medium.com/apache-zeppelin-stories/how-
>> apache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>
>> I hope all users and contributors understand Apache Zeppelin deeply.
>>
>> Regards,
>> Jongyoul
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by Jongyoul Lee <jo...@gmail.com>.
Hello DuyHai,
Thanks for the fixing the typo. I've fixed it.
Concerning the debugging, I think writing posts or updating wiki would be
better. I'm willing to write a simple post for that remote debugging.
On Tue, Oct 4, 2016 at 5:53 PM, DuyHai Doan <do...@gmail.com> wrote:
> Hello Jongyoul
>
> Excellent write up to demystify the execution life-cycle of a paragraph.
>
> There is a small typo in the blog: "At last, it has two words separated
> by comma, *getInterpreter* handles it as “%{group_name}.{interpreter_
> name}”"
>
> Instead of comma, shouldn't it be DOT (.) ?
>
> There is also another unknown details that may be interesting to expose:
> how are interpreter JVM launched ?
>
> InterpreterFactory
>
> private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
> Properties property, String interpreterSettingId) {
> int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
> String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
> int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
>
> RemoteInterpreter remoteInterpreter =
> new RemoteInterpreter(property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
> interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
> remoteInterpreterProcessListener, appEventListener);
> remoteInterpreter.addEnv(env);
>
> return new LazyOpenInterpreter(remoteInterpreter);
> }
>
>
> The code will call the bin/intepreter.sh (or interpreter.cmd if on
> Windows) to launch a new JVM for the interpreter.
>
> And depending on the type of interpreter, the command will be different:
>
> if [[ -n "${SPARK_SUBMIT}" ]]; then
> ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
> "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options
> "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
> else
> ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
> ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
> ${PORT} &
> fi
>
> There is a special case for Spark because we execute a SparkSubmit.
>
> This detail is important for contributors because when they need to DEBUG
> an interpreter code, they have to modify the interpreter.sh file to add an
> "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
>
> Most of people naively add this debug argument to the zeppelin.sh file but
> it only debugs the Zeppelin server itself, not the interpreter.
>
> Do you think we may create a page in the official doc for this ?
>
>
> On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
>> Hello devs and users,
>>
>> I've submitted a post into Apache Zeppelin Stories on medium.com:
>> https://medium.com/apache-zeppelin-stories/how-
>> apache-zeppelin-runs-a-paragraph-783a0a612ba9#.x9ym8fovv
>>
>> I hope all users and contributors understand Apache Zeppelin deeply.
>>
>> Regards,
>> Jongyoul
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: How Apache Zeppelin runs a paragraph
Posted by DuyHai Doan <do...@gmail.com>.
Hello Jongyoul
Excellent write up to demystify the execution life-cycle of a paragraph.
There is a small typo in the blog: "At last, it has two words separated by
comma, *getInterpreter* handles it as “%{group_name}.{interpreter_name}”"
Instead of comma, shouldn't it be DOT (.) ?
There is also another unknown details that may be interesting to expose:
how are interpreter JVM launched ?
InterpreterFactory
private Interpreter createRemoteRepl(String interpreterPath, String
noteId, String className,
Properties property, String interpreterSettingId) {
int connectTimeout =
conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" +
interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
RemoteInterpreter remoteInterpreter =
new RemoteInterpreter(property, noteId, className,
conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
The code will call the bin/intepreter.sh (or interpreter.cmd if on Windows)
to launch a new JVM for the interpreter.
And depending on the type of interpreter, the command will be different:
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options
"${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
${PORT} &
fi
There is a special case for Spark because we execute a SparkSubmit.
This detail is important for contributors because when they need to DEBUG
an interpreter code, they have to modify the interpreter.sh file to add an
"-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
Most of people naively add this debug argument to the zeppelin.sh file but
it only debugs the Zeppelin server itself, not the interpreter.
Do you think we may create a page in the official doc for this ?
On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
> Hello devs and users,
>
> I've submitted a post into Apache Zeppelin Stories on medium.com:
> https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-
> paragraph-783a0a612ba9#.x9ym8fovv
>
> I hope all users and contributors understand Apache Zeppelin deeply.
>
> Regards,
> Jongyoul
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: How Apache Zeppelin runs a paragraph
Posted by DuyHai Doan <do...@gmail.com>.
Hello Jongyoul
Excellent write up to demystify the execution life-cycle of a paragraph.
There is a small typo in the blog: "At last, it has two words separated by
comma, *getInterpreter* handles it as “%{group_name}.{interpreter_name}”"
Instead of comma, shouldn't it be DOT (.) ?
There is also another unknown details that may be interesting to expose:
how are interpreter JVM launched ?
InterpreterFactory
private Interpreter createRemoteRepl(String interpreterPath, String
noteId, String className,
Properties property, String interpreterSettingId) {
int connectTimeout =
conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" +
interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
RemoteInterpreter remoteInterpreter =
new RemoteInterpreter(property, noteId, className,
conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
The code will call the bin/intepreter.sh (or interpreter.cmd if on Windows)
to launch a new JVM for the interpreter.
And depending on the type of interpreter, the command will be different:
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options
"${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
${PORT} &
fi
There is a special case for Spark because we execute a SparkSubmit.
This detail is important for contributors because when they need to DEBUG
an interpreter code, they have to modify the interpreter.sh file to add an
"-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
Most of people naively add this debug argument to the zeppelin.sh file but
it only debugs the Zeppelin server itself, not the interpreter.
Do you think we may create a page in the official doc for this ?
On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jo...@gmail.com> wrote:
> Hello devs and users,
>
> I've submitted a post into Apache Zeppelin Stories on medium.com:
> https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-
> paragraph-783a0a612ba9#.x9ym8fovv
>
> I hope all users and contributors understand Apache Zeppelin deeply.
>
> Regards,
> Jongyoul
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>