You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Jhon Anderson Cardenas Diaz <jh...@gmail.com> on 2018/02/01 21:30:39 UTC
Extending SparkInterpreter functionality
Hello!
I'm a software developer and as part of a project I require to extend the
functionality of SparkInterpreter without modifying it. I need instead
create a new interpreter that extends it or wrap its functionality.
I also need the spark sub-interpreters to use my new custom interpreter,
but the problem comes here, because the spark sub-interpreters has a direct
dependency to spark interpreter as they use the class name of spark
interpreter to obtain its instance:
private SparkInterpreter getSparkInterpreter() {
...
Interpreter p =
getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
}
*Approach without modify apache zeppelin*
My current approach to solve is to create a SparkCustomInterpreter that
override the getClassName method as follows:
public class SparkCustomInterpreter extends SparkInterpreter {
...
@Override
public String getClassName() {
return SparkInterpreter.class.getName();
}
}
and put the new class name in the interpreter-setting.json file of spark:
[
{
"group": "spark",
"name": "spark",
"className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
...
"properties": {...}
}, ...
]
The problem with this approach is that when I run a paragraph it fails. In
general it fails because zeppelin uses both the class name of the instance
and the getClassName() method to access the instance, and that causes many
problems.
*Approaches modifying apache zeppelin*
There are two possible solutions related with the way in which the
sub-interpreters get the SparkInterpreter instance class, one is getting
the class name from a property:
private SparkInterpreter getSparkInterpreter() {
...
Interpreter p =
getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass",
SparkInterpreter.class.getName())* );
}
And the other possibility is to modify the method Interpreter.
getInterpreterInTheSameSessionByClassName(String) in order to return the
instance that whether has the same class name specified in the parameter or
which super class has the same class name specified in the parameter:
@ZeppelinApi
public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
synchronized (interpreterGroup) {
for (List<Interpreter> interpreters : interpreterGroup.values()) {
....
for (Interpreter intp : interpreters) {
if (intp.getClassName().equals(className) *||
intp.getClass().getSuperclass().getName().equals(className)*) {
interpreterFound = intp;
}
...
}
...
}
}
return null;
}
Either of the two solutions would involve the modification of apache
zeppelin code; do you think the change could be contributed to the
community?, or maybe do you realize some other approach to change the way
in which sub-interpreters of spark get the instance of spark interpreter?
Any information about it I'll be attempt.
Greetings
Jhon
Re: Extending SparkInterpreter functionality
Posted by Jhon Anderson Cardenas Diaz <jh...@gmail.com>.
Thanks for your replies Jeff,
We mention the deference between spark standalone and spark on yarn because
of the way in which spark UI URL is build by zeppelin, not for the the
possibility of connect multiple cluster:
*Spark UI URL on Yarn client mode in EMR*
In yarn client mode the access to spark is protected by YARN Application
Security (
https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html).
In order to access the spark UI we can not use the port 4040, instead we
have to use the port of the yarn proxy server, which is 20888.
Currently zeppelin supports a custom spark ui url by using the property
"zeppelin.spark.uiWebUrl", but is not enough an static value since proxy
server expects the application id of current spark context too, something
like:
*<proxy_server_addr>:20888/proxy/<applicationid>/jobs*
That's one of the reasons why we want to extends the spark interpreter.
Related links:
- https://forums.aws.amazon.com/thread.jspa?threadID=185677
-
https://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAPCxoxXvMC0vFDGw4XN14GEzG5uU=dY4fjDzCu6_o5Bc8cZCuQ@mail.gmail.com%3E
*Connecting multiple clusters*
For the feature of connect multiple clusters, the interpreter settings is a
static configuration and shared by the users, we would like to do that
dynamically depending on the user logged in zeppelin.
In general we would like zeppelin to allow the extension of the
SparkInterpreter component in order to use its basic characteristics and be
able to add more specific behaviors depending on each need.
Thank you,
Jhon
2018-02-02 2:05 GMT-05:00 Jeff Zhang <zj...@gmail.com>:
>
> >>> Same spark versions but multiple clusters. So based on logged in user,
> we may want to route to different spark cluster or even let user choose the
> spark he wants to connect to.
> If you use standalone, you can set `master` in interpreter setting for
> different standalone cluster. If using yarn, then you can set
> `HADOOP_CONF_DIR` for different yarn cluster.
>
>
> >>> Are you okay with us working on those tickets?
> Welcome contribution, I would love to help if you need.
>
>
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午2:51写道：
>
>> Hi Jeff,
>> #3 is not about different spark versions.
>>
>> Same spark versions but multiple clusters. So based on logged in user, we
>> may want to route to different spark cluster or even let user choose the
>> spark he wants to connect to.
>>
>> Will work with Jhon to create tickets on other #2.
>> What is the turn-around time for such tasks usually?
>> Are you okay with us working on those tickets?
>>
>> Maybe we can setup a meeting early Monday to discuss our proposals in
>> detail?
>>
>> Thanks
>> Ankit
>>
>> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>
>> 1) Spark UI which works differently on EMR than standalone, so that logic
>> will be in an interpreter specific to emr.
>> Could you create a ticket for that, and please add details of that ? I
>> don't know exactly what the difference between EMR and standalone, we can
>> expose api to allow customization if necessary.
>>
>>
>> 2) We want to add more metrics & logs in the interpreter, say number of
>> requests coming to the interpreter.
>> Could you create a ticket for that as well ? I think it is not
>> difficult to do that.
>>
>> 3) Ideally we will like to connect to different spark clusters in
>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>> It is possible now already, you can create different spark interpreter
>> for different spark clusters. e.g. you can create spark_16 for spark 1.6
>> and spark_22 for spark 2.2, and what you need to do is just setting
>> SPARK_HOME properly in their interpreter setting.
>>
>>
>> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>>
>>> This is exactly what we want Jeff! A hook to plug in our own
>>> interpreters.
>>> (I am on same team as Jhon btw)
>>>
>>> Right now there are too many concrete references and injecting stuff is
>>> not possible.
>>>
>>> Eg of customizations -
>>> 1) Spark UI which works differently on EMR than standalone, so that
>>> logic will be in an interpreter specific to emr.
>>> 2) We want to add more metrics & logs in the interpreter, say number of
>>> requests coming to the interpreter.
>>> 3) Ideally we will like to connect to different spark clusters in
>>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>>>
>>> Basically we want to add lot more flexibility.
>>>
>>> We are building a platform to cater to multiple clients. So, multiple
>>> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
>>> of that maintaining the security and privacy in a shared multi-tenant env
>>> will need all the flexibility we can get!
>>>
>>> Thanks
>>> Ankit
>>>
>>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>
>>> Hi Jhon,
>>>
>>> Do you mind to share what kind of custom function you want to add to
>>> spark interpreter ? One idea in my mind is that we could add extension
>>> point to the existing SparkInterpreter, and user can enhance
>>> SparkInterpreter via these extension point. That means we just open some
>>> interfaces and users can implement those interfaces, and just add their
>>> jars to spark interpreter folder.
>>>
>>>
>>>
>>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
>>> 上午5:30写道：
>>>
>>>> Hello!
>>>>
>>>> I'm a software developer and as part of a project I require to extend
>>>> the functionality of SparkInterpreter without modifying it. I need instead
>>>> create a new interpreter that extends it or wrap its functionality.
>>>>
>>>> I also need the spark sub-interpreters to use my new custom
>>>> interpreter, but the problem comes here, because the spark sub-interpreters
>>>> has a direct dependency to spark interpreter as they use the class name of
>>>> spark interpreter to obtain its instance:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>>
>>>> ...
>>>>
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>>
>>>> }
>>>>
>>>>
>>>> *Approach without modify apache zeppelin*
>>>>
>>>> My current approach to solve is to create a SparkCustomInterpreter that
>>>> override the getClassName method as follows:
>>>>
>>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>>> ...
>>>>
>>>> @Override
>>>> public String getClassName() {
>>>> return SparkInterpreter.class.getName();
>>>> }
>>>> }
>>>>
>>>>
>>>> and put the new class name in the interpreter-setting.json file of
>>>> spark:
>>>>
>>>> [
>>>> {
>>>> "group": "spark",
>>>> "name": "spark",
>>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>>> ...
>>>> "properties": {...}
>>>> }, ...
>>>> ]
>>>>
>>>>
>>>> The problem with this approach is that when I run a paragraph it fails.
>>>> In general it fails because zeppelin uses both the class name of the
>>>> instance and the getClassName() method to access the instance, and
>>>> that causes many problems.
>>>>
>>>> *Approaches modifying apache zeppelin*
>>>>
>>>> There are two possible solutions related with the way in which the
>>>> sub-interpreters get the SparkInterpreter instance class, one is
>>>> getting the class name from a property:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>>
>>>> ...
>>>>
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>>>
>>>> }
>>>>
>>>> And the other possibility is to modify the method Interpreter.
>>>> getInterpreterInTheSameSessionByClassName(String) in order to return
>>>> the instance that whether has the same class name specified in the
>>>> parameter or which super class has the same class name specified in the
>>>> parameter:
>>>>
>>>>
>>>> @ZeppelinApi
>>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>>> synchronized (interpreterGroup) {
>>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>>> ....
>>>> for (Interpreter intp : interpreters) {
>>>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>>>> interpreterFound = intp;
>>>> }
>>>>
>>>> ...
>>>> }
>>>>
>>>> ...
>>>> }
>>>> }
>>>> return null;
>>>> }
>>>>
>>>>
>>>> Either of the two solutions would involve the modification of apache
>>>> zeppelin code; do you think the change could be contributed to the
>>>> community?, or maybe do you realize some other approach to change the
>>>> way in which sub-interpreters of spark get the instance of spark interpreter
>>>> ?
>>>>
>>>> Any information about it I'll be attempt.
>>>>
>>>> Greetings
>>>>
>>>>
>>>> Jhon
>>>>
>>>
Re: Extending SparkInterpreter functionality
Posted by Jhon Anderson Cardenas Diaz <jh...@gmail.com>.
Thanks for your replies Jeff,
We mention the deference between spark standalone and spark on yarn because
of the way in which spark UI URL is build by zeppelin, not for the the
possibility of connect multiple cluster:
*Spark UI URL on Yarn client mode in EMR*
In yarn client mode the access to spark is protected by YARN Application
Security (
https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html).
In order to access the spark UI we can not use the port 4040, instead we
have to use the port of the yarn proxy server, which is 20888.
Currently zeppelin supports a custom spark ui url by using the property
"zeppelin.spark.uiWebUrl", but is not enough an static value since proxy
server expects the application id of current spark context too, something
like:
*<proxy_server_addr>:20888/proxy/<applicationid>/jobs*
That's one of the reasons why we want to extends the spark interpreter.
Related links:
- https://forums.aws.amazon.com/thread.jspa?threadID=185677
-
https://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAPCxoxXvMC0vFDGw4XN14GEzG5uU=dY4fjDzCu6_o5Bc8cZCuQ@mail.gmail.com%3E
*Connecting multiple clusters*
For the feature of connect multiple clusters, the interpreter settings is a
static configuration and shared by the users, we would like to do that
dynamically depending on the user logged in zeppelin.
In general we would like zeppelin to allow the extension of the
SparkInterpreter component in order to use its basic characteristics and be
able to add more specific behaviors depending on each need.
Thank you,
Jhon
2018-02-02 2:05 GMT-05:00 Jeff Zhang <zj...@gmail.com>:
>
> >>> Same spark versions but multiple clusters. So based on logged in user,
> we may want to route to different spark cluster or even let user choose the
> spark he wants to connect to.
> If you use standalone, you can set `master` in interpreter setting for
> different standalone cluster. If using yarn, then you can set
> `HADOOP_CONF_DIR` for different yarn cluster.
>
>
> >>> Are you okay with us working on those tickets?
> Welcome contribution, I would love to help if you need.
>
>
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午2:51写道：
>
>> Hi Jeff,
>> #3 is not about different spark versions.
>>
>> Same spark versions but multiple clusters. So based on logged in user, we
>> may want to route to different spark cluster or even let user choose the
>> spark he wants to connect to.
>>
>> Will work with Jhon to create tickets on other #2.
>> What is the turn-around time for such tasks usually?
>> Are you okay with us working on those tickets?
>>
>> Maybe we can setup a meeting early Monday to discuss our proposals in
>> detail?
>>
>> Thanks
>> Ankit
>>
>> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>
>> 1) Spark UI which works differently on EMR than standalone, so that logic
>> will be in an interpreter specific to emr.
>> Could you create a ticket for that, and please add details of that ? I
>> don't know exactly what the difference between EMR and standalone, we can
>> expose api to allow customization if necessary.
>>
>>
>> 2) We want to add more metrics & logs in the interpreter, say number of
>> requests coming to the interpreter.
>> Could you create a ticket for that as well ? I think it is not
>> difficult to do that.
>>
>> 3) Ideally we will like to connect to different spark clusters in
>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>> It is possible now already, you can create different spark interpreter
>> for different spark clusters. e.g. you can create spark_16 for spark 1.6
>> and spark_22 for spark 2.2, and what you need to do is just setting
>> SPARK_HOME properly in their interpreter setting.
>>
>>
>> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>>
>>> This is exactly what we want Jeff! A hook to plug in our own
>>> interpreters.
>>> (I am on same team as Jhon btw)
>>>
>>> Right now there are too many concrete references and injecting stuff is
>>> not possible.
>>>
>>> Eg of customizations -
>>> 1) Spark UI which works differently on EMR than standalone, so that
>>> logic will be in an interpreter specific to emr.
>>> 2) We want to add more metrics & logs in the interpreter, say number of
>>> requests coming to the interpreter.
>>> 3) Ideally we will like to connect to different spark clusters in
>>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>>>
>>> Basically we want to add lot more flexibility.
>>>
>>> We are building a platform to cater to multiple clients. So, multiple
>>> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
>>> of that maintaining the security and privacy in a shared multi-tenant env
>>> will need all the flexibility we can get!
>>>
>>> Thanks
>>> Ankit
>>>
>>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>
>>> Hi Jhon,
>>>
>>> Do you mind to share what kind of custom function you want to add to
>>> spark interpreter ? One idea in my mind is that we could add extension
>>> point to the existing SparkInterpreter, and user can enhance
>>> SparkInterpreter via these extension point. That means we just open some
>>> interfaces and users can implement those interfaces, and just add their
>>> jars to spark interpreter folder.
>>>
>>>
>>>
>>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
>>> 上午5:30写道：
>>>
>>>> Hello!
>>>>
>>>> I'm a software developer and as part of a project I require to extend
>>>> the functionality of SparkInterpreter without modifying it. I need instead
>>>> create a new interpreter that extends it or wrap its functionality.
>>>>
>>>> I also need the spark sub-interpreters to use my new custom
>>>> interpreter, but the problem comes here, because the spark sub-interpreters
>>>> has a direct dependency to spark interpreter as they use the class name of
>>>> spark interpreter to obtain its instance:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>>
>>>> ...
>>>>
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>>
>>>> }
>>>>
>>>>
>>>> *Approach without modify apache zeppelin*
>>>>
>>>> My current approach to solve is to create a SparkCustomInterpreter that
>>>> override the getClassName method as follows:
>>>>
>>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>>> ...
>>>>
>>>> @Override
>>>> public String getClassName() {
>>>> return SparkInterpreter.class.getName();
>>>> }
>>>> }
>>>>
>>>>
>>>> and put the new class name in the interpreter-setting.json file of
>>>> spark:
>>>>
>>>> [
>>>> {
>>>> "group": "spark",
>>>> "name": "spark",
>>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>>> ...
>>>> "properties": {...}
>>>> }, ...
>>>> ]
>>>>
>>>>
>>>> The problem with this approach is that when I run a paragraph it fails.
>>>> In general it fails because zeppelin uses both the class name of the
>>>> instance and the getClassName() method to access the instance, and
>>>> that causes many problems.
>>>>
>>>> *Approaches modifying apache zeppelin*
>>>>
>>>> There are two possible solutions related with the way in which the
>>>> sub-interpreters get the SparkInterpreter instance class, one is
>>>> getting the class name from a property:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>>
>>>> ...
>>>>
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>>>
>>>> }
>>>>
>>>> And the other possibility is to modify the method Interpreter.
>>>> getInterpreterInTheSameSessionByClassName(String) in order to return
>>>> the instance that whether has the same class name specified in the
>>>> parameter or which super class has the same class name specified in the
>>>> parameter:
>>>>
>>>>
>>>> @ZeppelinApi
>>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>>> synchronized (interpreterGroup) {
>>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>>> ....
>>>> for (Interpreter intp : interpreters) {
>>>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>>>> interpreterFound = intp;
>>>> }
>>>>
>>>> ...
>>>> }
>>>>
>>>> ...
>>>> }
>>>> }
>>>> return null;
>>>> }
>>>>
>>>>
>>>> Either of the two solutions would involve the modification of apache
>>>> zeppelin code; do you think the change could be contributed to the
>>>> community?, or maybe do you realize some other approach to change the
>>>> way in which sub-interpreters of spark get the instance of spark interpreter
>>>> ?
>>>>
>>>> Any information about it I'll be attempt.
>>>>
>>>> Greetings
>>>>
>>>>
>>>> Jhon
>>>>
>>>
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
>>> Same spark versions but multiple clusters. So based on logged in user,
we may want to route to different spark cluster or even let user choose the
spark he wants to connect to.
If you use standalone, you can set `master` in interpreter setting for
different standalone cluster. If using yarn, then you can set
`HADOOP_CONF_DIR` for different yarn cluster.
>>> Are you okay with us working on those tickets?
Welcome contribution, I would love to help if you need.
Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午2:51写道：
> Hi Jeff,
> #3 is not about different spark versions.
>
> Same spark versions but multiple clusters. So based on logged in user, we
> may want to route to different spark cluster or even let user choose the
> spark he wants to connect to.
>
> Will work with Jhon to create tickets on other #2.
> What is the turn-around time for such tasks usually?
> Are you okay with us working on those tickets?
>
> Maybe we can setup a meeting early Monday to discuss our proposals in
> detail?
>
> Thanks
> Ankit
>
> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> 1) Spark UI which works differently on EMR than standalone, so that logic
> will be in an interpreter specific to emr.
> Could you create a ticket for that, and please add details of that ? I
> don't know exactly what the difference between EMR and standalone, we can
> expose api to allow customization if necessary.
>
>
> 2) We want to add more metrics & logs in the interpreter, say number of
> requests coming to the interpreter.
> Could you create a ticket for that as well ? I think it is not
> difficult to do that.
>
> 3) Ideally we will like to connect to different spark clusters in
> spark-submit and not tie to one which happens on Zeppelin startup right now.
> It is possible now already, you can create different spark interpreter for
> different spark clusters. e.g. you can create spark_16 for spark 1.6 and
> spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME
> properly in their interpreter setting.
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>
>> This is exactly what we want Jeff! A hook to plug in our own interpreters.
>> (I am on same team as Jhon btw)
>>
>> Right now there are too many concrete references and injecting stuff is
>> not possible.
>>
>> Eg of customizations -
>> 1) Spark UI which works differently on EMR than standalone, so that logic
>> will be in an interpreter specific to emr.
>> 2) We want to add more metrics & logs in the interpreter, say number of
>> requests coming to the interpreter.
>> 3) Ideally we will like to connect to different spark clusters in
>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>>
>> Basically we want to add lot more flexibility.
>>
>> We are building a platform to cater to multiple clients. So, multiple
>> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
>> of that maintaining the security and privacy in a shared multi-tenant env
>> will need all the flexibility we can get!
>>
>> Thanks
>> Ankit
>>
>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>
>> Hi Jhon,
>>
>> Do you mind to share what kind of custom function you want to add to
>> spark interpreter ? One idea in my mind is that we could add extension
>> point to the existing SparkInterpreter, and user can enhance
>> SparkInterpreter via these extension point. That means we just open some
>> interfaces and users can implement those interfaces, and just add their
>> jars to spark interpreter folder.
>>
>>
>>
>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
>> 上午5:30写道：
>>
>>> Hello!
>>>
>>> I'm a software developer and as part of a project I require to extend
>>> the functionality of SparkInterpreter without modifying it. I need instead
>>> create a new interpreter that extends it or wrap its functionality.
>>>
>>> I also need the spark sub-interpreters to use my new custom
>>> interpreter, but the problem comes here, because the spark sub-interpreters
>>> has a direct dependency to spark interpreter as they use the class name of
>>> spark interpreter to obtain its instance:
>>>
>>>
>>> private SparkInterpreter getSparkInterpreter() {
>>>
>>> ...
>>>
>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>
>>> }
>>>
>>>
>>> *Approach without modify apache zeppelin*
>>>
>>> My current approach to solve is to create a SparkCustomInterpreter that
>>> override the getClassName method as follows:
>>>
>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>> ...
>>>
>>> @Override
>>> public String getClassName() {
>>> return SparkInterpreter.class.getName();
>>> }
>>> }
>>>
>>>
>>> and put the new class name in the interpreter-setting.json file of spark:
>>>
>>> [
>>> {
>>> "group": "spark",
>>> "name": "spark",
>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>> ...
>>> "properties": {...}
>>> }, ...
>>> ]
>>>
>>>
>>> The problem with this approach is that when I run a paragraph it fails.
>>> In general it fails because zeppelin uses both the class name of the
>>> instance and the getClassName() method to access the instance, and that
>>> causes many problems.
>>>
>>> *Approaches modifying apache zeppelin*
>>>
>>> There are two possible solutions related with the way in which the
>>> sub-interpreters get the SparkInterpreter instance class, one is
>>> getting the class name from a property:
>>>
>>>
>>> private SparkInterpreter getSparkInterpreter() {
>>>
>>> ...
>>>
>>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>>
>>> }
>>>
>>> And the other possibility is to modify the method
>>> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
>>> return the instance that whether has the same class name specified in the
>>> parameter or which super class has the same class name specified in the
>>> parameter:
>>>
>>>
>>> @ZeppelinApi
>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>> synchronized (interpreterGroup) {
>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>> ....
>>> for (Interpreter intp : interpreters) {
>>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>>> interpreterFound = intp;
>>> }
>>>
>>> ...
>>> }
>>>
>>> ...
>>> }
>>> }
>>> return null;
>>> }
>>>
>>>
>>> Either of the two solutions would involve the modification of apache
>>> zeppelin code; do you think the change could be contributed to the
>>> community?, or maybe do you realize some other approach to change the
>>> way in which sub-interpreters of spark get the instance of spark interpreter
>>> ?
>>>
>>> Any information about it I'll be attempt.
>>>
>>> Greetings
>>>
>>>
>>> Jhon
>>>
>>
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
>>> Same spark versions but multiple clusters. So based on logged in user,
we may want to route to different spark cluster or even let user choose the
spark he wants to connect to.
If you use standalone, you can set `master` in interpreter setting for
different standalone cluster. If using yarn, then you can set
`HADOOP_CONF_DIR` for different yarn cluster.
>>> Are you okay with us working on those tickets?
Welcome contribution, I would love to help if you need.
Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午2:51写道：
> Hi Jeff,
> #3 is not about different spark versions.
>
> Same spark versions but multiple clusters. So based on logged in user, we
> may want to route to different spark cluster or even let user choose the
> spark he wants to connect to.
>
> Will work with Jhon to create tickets on other #2.
> What is the turn-around time for such tasks usually?
> Are you okay with us working on those tickets?
>
> Maybe we can setup a meeting early Monday to discuss our proposals in
> detail?
>
> Thanks
> Ankit
>
> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> 1) Spark UI which works differently on EMR than standalone, so that logic
> will be in an interpreter specific to emr.
> Could you create a ticket for that, and please add details of that ? I
> don't know exactly what the difference between EMR and standalone, we can
> expose api to allow customization if necessary.
>
>
> 2) We want to add more metrics & logs in the interpreter, say number of
> requests coming to the interpreter.
> Could you create a ticket for that as well ? I think it is not
> difficult to do that.
>
> 3) Ideally we will like to connect to different spark clusters in
> spark-submit and not tie to one which happens on Zeppelin startup right now.
> It is possible now already, you can create different spark interpreter for
> different spark clusters. e.g. you can create spark_16 for spark 1.6 and
> spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME
> properly in their interpreter setting.
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>
>> This is exactly what we want Jeff! A hook to plug in our own interpreters.
>> (I am on same team as Jhon btw)
>>
>> Right now there are too many concrete references and injecting stuff is
>> not possible.
>>
>> Eg of customizations -
>> 1) Spark UI which works differently on EMR than standalone, so that logic
>> will be in an interpreter specific to emr.
>> 2) We want to add more metrics & logs in the interpreter, say number of
>> requests coming to the interpreter.
>> 3) Ideally we will like to connect to different spark clusters in
>> spark-submit and not tie to one which happens on Zeppelin startup right now.
>>
>> Basically we want to add lot more flexibility.
>>
>> We are building a platform to cater to multiple clients. So, multiple
>> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
>> of that maintaining the security and privacy in a shared multi-tenant env
>> will need all the flexibility we can get!
>>
>> Thanks
>> Ankit
>>
>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>
>> Hi Jhon,
>>
>> Do you mind to share what kind of custom function you want to add to
>> spark interpreter ? One idea in my mind is that we could add extension
>> point to the existing SparkInterpreter, and user can enhance
>> SparkInterpreter via these extension point. That means we just open some
>> interfaces and users can implement those interfaces, and just add their
>> jars to spark interpreter folder.
>>
>>
>>
>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
>> 上午5:30写道：
>>
>>> Hello!
>>>
>>> I'm a software developer and as part of a project I require to extend
>>> the functionality of SparkInterpreter without modifying it. I need instead
>>> create a new interpreter that extends it or wrap its functionality.
>>>
>>> I also need the spark sub-interpreters to use my new custom
>>> interpreter, but the problem comes here, because the spark sub-interpreters
>>> has a direct dependency to spark interpreter as they use the class name of
>>> spark interpreter to obtain its instance:
>>>
>>>
>>> private SparkInterpreter getSparkInterpreter() {
>>>
>>> ...
>>>
>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>
>>> }
>>>
>>>
>>> *Approach without modify apache zeppelin*
>>>
>>> My current approach to solve is to create a SparkCustomInterpreter that
>>> override the getClassName method as follows:
>>>
>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>> ...
>>>
>>> @Override
>>> public String getClassName() {
>>> return SparkInterpreter.class.getName();
>>> }
>>> }
>>>
>>>
>>> and put the new class name in the interpreter-setting.json file of spark:
>>>
>>> [
>>> {
>>> "group": "spark",
>>> "name": "spark",
>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>> ...
>>> "properties": {...}
>>> }, ...
>>> ]
>>>
>>>
>>> The problem with this approach is that when I run a paragraph it fails.
>>> In general it fails because zeppelin uses both the class name of the
>>> instance and the getClassName() method to access the instance, and that
>>> causes many problems.
>>>
>>> *Approaches modifying apache zeppelin*
>>>
>>> There are two possible solutions related with the way in which the
>>> sub-interpreters get the SparkInterpreter instance class, one is
>>> getting the class name from a property:
>>>
>>>
>>> private SparkInterpreter getSparkInterpreter() {
>>>
>>> ...
>>>
>>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>>
>>> }
>>>
>>> And the other possibility is to modify the method
>>> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
>>> return the instance that whether has the same class name specified in the
>>> parameter or which super class has the same class name specified in the
>>> parameter:
>>>
>>>
>>> @ZeppelinApi
>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>> synchronized (interpreterGroup) {
>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>> ....
>>> for (Interpreter intp : interpreters) {
>>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>>> interpreterFound = intp;
>>> }
>>>
>>> ...
>>> }
>>>
>>> ...
>>> }
>>> }
>>> return null;
>>> }
>>>
>>>
>>> Either of the two solutions would involve the modification of apache
>>> zeppelin code; do you think the change could be contributed to the
>>> community?, or maybe do you realize some other approach to change the
>>> way in which sub-interpreters of spark get the instance of spark interpreter
>>> ?
>>>
>>> Any information about it I'll be attempt.
>>>
>>> Greetings
>>>
>>>
>>> Jhon
>>>
>>
Re: Extending SparkInterpreter functionality
Posted by Ankit Jain <an...@gmail.com>.
Hi Jeff,
#3 is not about different spark versions.
Same spark versions but multiple clusters. So based on logged in user, we may want to route to different spark cluster or even let user choose the spark he wants to connect to.
Will work with Jhon to create tickets on other #2.
What is the turn-around time for such tasks usually?
Are you okay with us working on those tickets?
Maybe we can setup a meeting early Monday to discuss our proposals in detail?
Thanks
Ankit
> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> 1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
> Could you create a ticket for that, and please add details of that ? I don't know exactly what the difference between EMR and standalone, we can expose api to allow customization if necessary.
>
>
> 2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
> Could you create a ticket for that as well ? I think it is not difficult to do that.
>
> 3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
> It is possible now already, you can create different spark interpreter for different spark clusters. e.g. you can create spark_16 for spark 1.6 and spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME properly in their interpreter setting.
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>> This is exactly what we want Jeff! A hook to plug in our own interpreters.
>> (I am on same team as Jhon btw)
>>
>> Right now there are too many concrete references and injecting stuff is not possible.
>>
>> Eg of customizations -
>> 1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
>> 2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
>> 3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
>>
>> Basically we want to add lot more flexibility.
>>
>> We are building a platform to cater to multiple clients. So, multiple Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top of that maintaining the security and privacy in a shared multi-tenant env will need all the flexibility we can get!
>>
>> Thanks
>> Ankit
>>
>>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>
>>> Hi Jhon,
>>>
>>> Do you mind to share what kind of custom function you want to add to spark interpreter ? One idea in my mind is that we could add extension point to the existing SparkInterpreter, and user can enhance SparkInterpreter via these extension point. That means we just open some interfaces and users can implement those interfaces, and just add their jars to spark interpreter folder.
>>>
>>>
>>>
>>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
>>>> Hello!
>>>>
>>>> I'm a software developer and as part of a project I require to extend the functionality of SparkInterpreter without modifying it. I need instead create a new interpreter that extends it or wrap its functionality.
>>>>
>>>> I also need the spark sub-interpreters to use my new custom interpreter, but the problem comes here, because the spark sub-interpreters has a direct dependency to spark interpreter as they use the class name of spark interpreter to obtain its instance:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>> ...
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>> }
>>>>
>>>>
>>>> Approach without modify apache zeppelin
>>>>
>>>> My current approach to solve is to create a SparkCustomInterpreter that override the getClassName method as follows:
>>>>
>>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>>> ...
>>>>
>>>> @Override
>>>> public String getClassName() {
>>>> return SparkInterpreter.class.getName();
>>>> }
>>>> }
>>>>
>>>> and put the new class name in the interpreter-setting.json file of spark:
>>>>
>>>> [
>>>> {
>>>> "group": "spark",
>>>> "name": "spark",
>>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>>> ...
>>>> "properties": {...}
>>>> }, ...
>>>> ]
>>>>
>>>> The problem with this approach is that when I run a paragraph it fails. In general it fails because zeppelin uses both the class name of the instance and the getClassName() method to access the instance, and that causes many problems.
>>>>
>>>> Approaches modifying apache zeppelin
>>>>
>>>> There are two possible solutions related with the way in which the sub-interpreters get the SparkInterpreter instance class, one is getting the class name from a property:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>> ...
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName()) );
>>>> }
>>>> And the other possibility is to modify the method Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to return the instance that whether has the same class name specified in the parameter or which super class has the same class name specified in the parameter:
>>>>
>>>>
>>>> @ZeppelinApi
>>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>>> synchronized (interpreterGroup) {
>>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>>> ....
>>>> for (Interpreter intp : interpreters) {
>>>> if (intp.getClassName().equals(className) || intp.getClass().getSuperclass().getName().equals(className)) {
>>>> interpreterFound = intp;
>>>> }
>>>>
>>>> ...
>>>> }
>>>>
>>>> ...
>>>> }
>>>> }
>>>> return null;
>>>> }
>>>>
>>>> Either of the two solutions would involve the modification of apache zeppelin code; do you think the change could be contributed to the community?, or maybe do you realize some other approach to change the way in which sub-interpreters of spark get the instance of spark interpreter?
>>>>
>>>> Any information about it I'll be attempt.
>>>>
>>>> Greetings
>>>>
>>>>
>>>> Jhon
Re: Extending SparkInterpreter functionality
Posted by Ankit Jain <an...@gmail.com>.
Hi Jeff,
#3 is not about different spark versions.
Same spark versions but multiple clusters. So based on logged in user, we may want to route to different spark cluster or even let user choose the spark he wants to connect to.
Will work with Jhon to create tickets on other #2.
What is the turn-around time for such tasks usually?
Are you okay with us working on those tickets?
Maybe we can setup a meeting early Monday to discuss our proposals in detail?
Thanks
Ankit
> On Feb 1, 2018, at 10:15 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> 1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
> Could you create a ticket for that, and please add details of that ? I don't know exactly what the difference between EMR and standalone, we can expose api to allow customization if necessary.
>
>
> 2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
> Could you create a ticket for that as well ? I think it is not difficult to do that.
>
> 3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
> It is possible now already, you can create different spark interpreter for different spark clusters. e.g. you can create spark_16 for spark 1.6 and spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME properly in their interpreter setting.
>
>
> Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
>> This is exactly what we want Jeff! A hook to plug in our own interpreters.
>> (I am on same team as Jhon btw)
>>
>> Right now there are too many concrete references and injecting stuff is not possible.
>>
>> Eg of customizations -
>> 1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
>> 2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
>> 3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
>>
>> Basically we want to add lot more flexibility.
>>
>> We are building a platform to cater to multiple clients. So, multiple Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top of that maintaining the security and privacy in a shared multi-tenant env will need all the flexibility we can get!
>>
>> Thanks
>> Ankit
>>
>>> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>
>>> Hi Jhon,
>>>
>>> Do you mind to share what kind of custom function you want to add to spark interpreter ? One idea in my mind is that we could add extension point to the existing SparkInterpreter, and user can enhance SparkInterpreter via these extension point. That means we just open some interfaces and users can implement those interfaces, and just add their jars to spark interpreter folder.
>>>
>>>
>>>
>>> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
>>>> Hello!
>>>>
>>>> I'm a software developer and as part of a project I require to extend the functionality of SparkInterpreter without modifying it. I need instead create a new interpreter that extends it or wrap its functionality.
>>>>
>>>> I also need the spark sub-interpreters to use my new custom interpreter, but the problem comes here, because the spark sub-interpreters has a direct dependency to spark interpreter as they use the class name of spark interpreter to obtain its instance:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>> ...
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>>> }
>>>>
>>>>
>>>> Approach without modify apache zeppelin
>>>>
>>>> My current approach to solve is to create a SparkCustomInterpreter that override the getClassName method as follows:
>>>>
>>>> public class SparkCustomInterpreter extends SparkInterpreter {
>>>> ...
>>>>
>>>> @Override
>>>> public String getClassName() {
>>>> return SparkInterpreter.class.getName();
>>>> }
>>>> }
>>>>
>>>> and put the new class name in the interpreter-setting.json file of spark:
>>>>
>>>> [
>>>> {
>>>> "group": "spark",
>>>> "name": "spark",
>>>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>>>> ...
>>>> "properties": {...}
>>>> }, ...
>>>> ]
>>>>
>>>> The problem with this approach is that when I run a paragraph it fails. In general it fails because zeppelin uses both the class name of the instance and the getClassName() method to access the instance, and that causes many problems.
>>>>
>>>> Approaches modifying apache zeppelin
>>>>
>>>> There are two possible solutions related with the way in which the sub-interpreters get the SparkInterpreter instance class, one is getting the class name from a property:
>>>>
>>>>
>>>> private SparkInterpreter getSparkInterpreter() {
>>>> ...
>>>> Interpreter p = getInterpreterInTheSameSessionByClassName(property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName()) );
>>>> }
>>>> And the other possibility is to modify the method Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to return the instance that whether has the same class name specified in the parameter or which super class has the same class name specified in the parameter:
>>>>
>>>>
>>>> @ZeppelinApi
>>>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>>>> synchronized (interpreterGroup) {
>>>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>>>> ....
>>>> for (Interpreter intp : interpreters) {
>>>> if (intp.getClassName().equals(className) || intp.getClass().getSuperclass().getName().equals(className)) {
>>>> interpreterFound = intp;
>>>> }
>>>>
>>>> ...
>>>> }
>>>>
>>>> ...
>>>> }
>>>> }
>>>> return null;
>>>> }
>>>>
>>>> Either of the two solutions would involve the modification of apache zeppelin code; do you think the change could be contributed to the community?, or maybe do you realize some other approach to change the way in which sub-interpreters of spark get the instance of spark interpreter?
>>>>
>>>> Any information about it I'll be attempt.
>>>>
>>>> Greetings
>>>>
>>>>
>>>> Jhon
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
1) Spark UI which works differently on EMR than standalone, so that logic
will be in an interpreter specific to emr.
Could you create a ticket for that, and please add details of that ? I
don't know exactly what the difference between EMR and standalone, we can
expose api to allow customization if necessary.
2) We want to add more metrics & logs in the interpreter, say number of
requests coming to the interpreter.
Could you create a ticket for that as well ? I think it is not difficult
to do that.
3) Ideally we will like to connect to different spark clusters in
spark-submit and not tie to one which happens on Zeppelin startup right now.
It is possible now already, you can create different spark interpreter for
different spark clusters. e.g. you can create spark_16 for spark 1.6 and
spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME
properly in their interpreter setting.
Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
> This is exactly what we want Jeff! A hook to plug in our own interpreters.
> (I am on same team as Jhon btw)
>
> Right now there are too many concrete references and injecting stuff is
> not possible.
>
> Eg of customizations -
> 1) Spark UI which works differently on EMR than standalone, so that logic
> will be in an interpreter specific to emr.
> 2) We want to add more metrics & logs in the interpreter, say number of
> requests coming to the interpreter.
> 3) Ideally we will like to connect to different spark clusters in
> spark-submit and not tie to one which happens on Zeppelin startup right now.
>
> Basically we want to add lot more flexibility.
>
> We are building a platform to cater to multiple clients. So, multiple
> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
> of that maintaining the security and privacy in a shared multi-tenant env
> will need all the flexibility we can get!
>
> Thanks
> Ankit
>
> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Hi Jhon,
>
> Do you mind to share what kind of custom function you want to add to spark
> interpreter ? One idea in my mind is that we could add extension point to
> the existing SparkInterpreter, and user can enhance SparkInterpreter via
> these extension point. That means we just open some interfaces and users
> can implement those interfaces, and just add their jars to spark
> interpreter folder.
>
>
>
> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
> 上午5:30写道：
>
>> Hello!
>>
>> I'm a software developer and as part of a project I require to extend the
>> functionality of SparkInterpreter without modifying it. I need instead
>> create a new interpreter that extends it or wrap its functionality.
>>
>> I also need the spark sub-interpreters to use my new custom interpreter,
>> but the problem comes here, because the spark sub-interpreters has a
>> direct dependency to spark interpreter as they use the class name of spark
>> interpreter to obtain its instance:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>>
>> ...
>>
>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>
>> }
>>
>>
>> *Approach without modify apache zeppelin*
>>
>> My current approach to solve is to create a SparkCustomInterpreter that
>> override the getClassName method as follows:
>>
>> public class SparkCustomInterpreter extends SparkInterpreter {
>> ...
>>
>> @Override
>> public String getClassName() {
>> return SparkInterpreter.class.getName();
>> }
>> }
>>
>>
>> and put the new class name in the interpreter-setting.json file of spark:
>>
>> [
>> {
>> "group": "spark",
>> "name": "spark",
>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>> ...
>> "properties": {...}
>> }, ...
>> ]
>>
>>
>> The problem with this approach is that when I run a paragraph it fails.
>> In general it fails because zeppelin uses both the class name of the
>> instance and the getClassName() method to access the instance, and that
>> causes many problems.
>>
>> *Approaches modifying apache zeppelin*
>>
>> There are two possible solutions related with the way in which the
>> sub-interpreters get the SparkInterpreter instance class, one is getting
>> the class name from a property:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>>
>> ...
>>
>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>
>> }
>>
>> And the other possibility is to modify the method
>> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
>> return the instance that whether has the same class name specified in the
>> parameter or which super class has the same class name specified in the
>> parameter:
>>
>>
>> @ZeppelinApi
>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>> synchronized (interpreterGroup) {
>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>> ....
>> for (Interpreter intp : interpreters) {
>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>> interpreterFound = intp;
>> }
>>
>> ...
>> }
>>
>> ...
>> }
>> }
>> return null;
>> }
>>
>>
>> Either of the two solutions would involve the modification of apache
>> zeppelin code; do you think the change could be contributed to the
>> community?, or maybe do you realize some other approach to change the
>> way in which sub-interpreters of spark get the instance of spark interpreter
>> ?
>>
>> Any information about it I'll be attempt.
>>
>> Greetings
>>
>>
>> Jhon
>>
>
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
1) Spark UI which works differently on EMR than standalone, so that logic
will be in an interpreter specific to emr.
Could you create a ticket for that, and please add details of that ? I
don't know exactly what the difference between EMR and standalone, we can
expose api to allow customization if necessary.
2) We want to add more metrics & logs in the interpreter, say number of
requests coming to the interpreter.
Could you create a ticket for that as well ? I think it is not difficult
to do that.
3) Ideally we will like to connect to different spark clusters in
spark-submit and not tie to one which happens on Zeppelin startup right now.
It is possible now already, you can create different spark interpreter for
different spark clusters. e.g. you can create spark_16 for spark 1.6 and
spark_22 for spark 2.2, and what you need to do is just setting SPARK_HOME
properly in their interpreter setting.
Ankit Jain <an...@gmail.com>于2018年2月2日周五 下午1:36写道：
> This is exactly what we want Jeff! A hook to plug in our own interpreters.
> (I am on same team as Jhon btw)
>
> Right now there are too many concrete references and injecting stuff is
> not possible.
>
> Eg of customizations -
> 1) Spark UI which works differently on EMR than standalone, so that logic
> will be in an interpreter specific to emr.
> 2) We want to add more metrics & logs in the interpreter, say number of
> requests coming to the interpreter.
> 3) Ideally we will like to connect to different spark clusters in
> spark-submit and not tie to one which happens on Zeppelin startup right now.
>
> Basically we want to add lot more flexibility.
>
> We are building a platform to cater to multiple clients. So, multiple
> Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top
> of that maintaining the security and privacy in a shared multi-tenant env
> will need all the flexibility we can get!
>
> Thanks
> Ankit
>
> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Hi Jhon,
>
> Do you mind to share what kind of custom function you want to add to spark
> interpreter ? One idea in my mind is that we could add extension point to
> the existing SparkInterpreter, and user can enhance SparkInterpreter via
> these extension point. That means we just open some interfaces and users
> can implement those interfaces, and just add their jars to spark
> interpreter folder.
>
>
>
> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五
> 上午5:30写道：
>
>> Hello!
>>
>> I'm a software developer and as part of a project I require to extend the
>> functionality of SparkInterpreter without modifying it. I need instead
>> create a new interpreter that extends it or wrap its functionality.
>>
>> I also need the spark sub-interpreters to use my new custom interpreter,
>> but the problem comes here, because the spark sub-interpreters has a
>> direct dependency to spark interpreter as they use the class name of spark
>> interpreter to obtain its instance:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>>
>> ...
>>
>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>>
>> }
>>
>>
>> *Approach without modify apache zeppelin*
>>
>> My current approach to solve is to create a SparkCustomInterpreter that
>> override the getClassName method as follows:
>>
>> public class SparkCustomInterpreter extends SparkInterpreter {
>> ...
>>
>> @Override
>> public String getClassName() {
>> return SparkInterpreter.class.getName();
>> }
>> }
>>
>>
>> and put the new class name in the interpreter-setting.json file of spark:
>>
>> [
>> {
>> "group": "spark",
>> "name": "spark",
>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>> ...
>> "properties": {...}
>> }, ...
>> ]
>>
>>
>> The problem with this approach is that when I run a paragraph it fails.
>> In general it fails because zeppelin uses both the class name of the
>> instance and the getClassName() method to access the instance, and that
>> causes many problems.
>>
>> *Approaches modifying apache zeppelin*
>>
>> There are two possible solutions related with the way in which the
>> sub-interpreters get the SparkInterpreter instance class, one is getting
>> the class name from a property:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>>
>> ...
>>
>> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>>
>> }
>>
>> And the other possibility is to modify the method
>> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
>> return the instance that whether has the same class name specified in the
>> parameter or which super class has the same class name specified in the
>> parameter:
>>
>>
>> @ZeppelinApi
>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>> synchronized (interpreterGroup) {
>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>> ....
>> for (Interpreter intp : interpreters) {
>> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
>> interpreterFound = intp;
>> }
>>
>> ...
>> }
>>
>> ...
>> }
>> }
>> return null;
>> }
>>
>>
>> Either of the two solutions would involve the modification of apache
>> zeppelin code; do you think the change could be contributed to the
>> community?, or maybe do you realize some other approach to change the
>> way in which sub-interpreters of spark get the instance of spark interpreter
>> ?
>>
>> Any information about it I'll be attempt.
>>
>> Greetings
>>
>>
>> Jhon
>>
>
Re: Extending SparkInterpreter functionality
Posted by Ankit Jain <an...@gmail.com>.
This is exactly what we want Jeff! A hook to plug in our own interpreters.
(I am on same team as Jhon btw)
Right now there are too many concrete references and injecting stuff is not possible.
Eg of customizations -
1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
Basically we want to add lot more flexibility.
We are building a platform to cater to multiple clients. So, multiple Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top of that maintaining the security and privacy in a shared multi-tenant env will need all the flexibility we can get!
Thanks
Ankit
> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Hi Jhon,
>
> Do you mind to share what kind of custom function you want to add to spark interpreter ? One idea in my mind is that we could add extension point to the existing SparkInterpreter, and user can enhance SparkInterpreter via these extension point. That means we just open some interfaces and users can implement those interfaces, and just add their jars to spark interpreter folder.
>
>
>
> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
>> Hello!
>>
>> I'm a software developer and as part of a project I require to extend the functionality of SparkInterpreter without modifying it. I need instead create a new interpreter that extends it or wrap its functionality.
>>
>> I also need the spark sub-interpreters to use my new custom interpreter, but the problem comes here, because the spark sub-interpreters has a direct dependency to spark interpreter as they use the class name of spark interpreter to obtain its instance:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>> ...
>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>> }
>>
>>
>> Approach without modify apache zeppelin
>>
>> My current approach to solve is to create a SparkCustomInterpreter that override the getClassName method as follows:
>>
>> public class SparkCustomInterpreter extends SparkInterpreter {
>> ...
>>
>> @Override
>> public String getClassName() {
>> return SparkInterpreter.class.getName();
>> }
>> }
>>
>> and put the new class name in the interpreter-setting.json file of spark:
>>
>> [
>> {
>> "group": "spark",
>> "name": "spark",
>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>> ...
>> "properties": {...}
>> }, ...
>> ]
>>
>> The problem with this approach is that when I run a paragraph it fails. In general it fails because zeppelin uses both the class name of the instance and the getClassName() method to access the instance, and that causes many problems.
>>
>> Approaches modifying apache zeppelin
>>
>> There are two possible solutions related with the way in which the sub-interpreters get the SparkInterpreter instance class, one is getting the class name from a property:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>> ...
>> Interpreter p = getInterpreterInTheSameSessionByClassName(property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName()) );
>> }
>> And the other possibility is to modify the method Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to return the instance that whether has the same class name specified in the parameter or which super class has the same class name specified in the parameter:
>>
>>
>> @ZeppelinApi
>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>> synchronized (interpreterGroup) {
>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>> ....
>> for (Interpreter intp : interpreters) {
>> if (intp.getClassName().equals(className) || intp.getClass().getSuperclass().getName().equals(className)) {
>> interpreterFound = intp;
>> }
>>
>> ...
>> }
>>
>> ...
>> }
>> }
>> return null;
>> }
>>
>> Either of the two solutions would involve the modification of apache zeppelin code; do you think the change could be contributed to the community?, or maybe do you realize some other approach to change the way in which sub-interpreters of spark get the instance of spark interpreter?
>>
>> Any information about it I'll be attempt.
>>
>> Greetings
>>
>>
>> Jhon
Re: Extending SparkInterpreter functionality
Posted by Ankit Jain <an...@gmail.com>.
This is exactly what we want Jeff! A hook to plug in our own interpreters.
(I am on same team as Jhon btw)
Right now there are too many concrete references and injecting stuff is not possible.
Eg of customizations -
1) Spark UI which works differently on EMR than standalone, so that logic will be in an interpreter specific to emr.
2) We want to add more metrics & logs in the interpreter, say number of requests coming to the interpreter.
3) Ideally we will like to connect to different spark clusters in spark-submit and not tie to one which happens on Zeppelin startup right now.
Basically we want to add lot more flexibility.
We are building a platform to cater to multiple clients. So, multiple Zeppelin instances, multiple spark clusters, multiple Spark UIs and on top of that maintaining the security and privacy in a shared multi-tenant env will need all the flexibility we can get!
Thanks
Ankit
> On Feb 1, 2018, at 7:51 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>
> Hi Jhon,
>
> Do you mind to share what kind of custom function you want to add to spark interpreter ? One idea in my mind is that we could add extension point to the existing SparkInterpreter, and user can enhance SparkInterpreter via these extension point. That means we just open some interfaces and users can implement those interfaces, and just add their jars to spark interpreter folder.
>
>
>
> Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
>> Hello!
>>
>> I'm a software developer and as part of a project I require to extend the functionality of SparkInterpreter without modifying it. I need instead create a new interpreter that extends it or wrap its functionality.
>>
>> I also need the spark sub-interpreters to use my new custom interpreter, but the problem comes here, because the spark sub-interpreters has a direct dependency to spark interpreter as they use the class name of spark interpreter to obtain its instance:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>> ...
>> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>> }
>>
>>
>> Approach without modify apache zeppelin
>>
>> My current approach to solve is to create a SparkCustomInterpreter that override the getClassName method as follows:
>>
>> public class SparkCustomInterpreter extends SparkInterpreter {
>> ...
>>
>> @Override
>> public String getClassName() {
>> return SparkInterpreter.class.getName();
>> }
>> }
>>
>> and put the new class name in the interpreter-setting.json file of spark:
>>
>> [
>> {
>> "group": "spark",
>> "name": "spark",
>> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
>> ...
>> "properties": {...}
>> }, ...
>> ]
>>
>> The problem with this approach is that when I run a paragraph it fails. In general it fails because zeppelin uses both the class name of the instance and the getClassName() method to access the instance, and that causes many problems.
>>
>> Approaches modifying apache zeppelin
>>
>> There are two possible solutions related with the way in which the sub-interpreters get the SparkInterpreter instance class, one is getting the class name from a property:
>>
>>
>> private SparkInterpreter getSparkInterpreter() {
>> ...
>> Interpreter p = getInterpreterInTheSameSessionByClassName(property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName()) );
>> }
>> And the other possibility is to modify the method Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to return the instance that whether has the same class name specified in the parameter or which super class has the same class name specified in the parameter:
>>
>>
>> @ZeppelinApi
>> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
>> synchronized (interpreterGroup) {
>> for (List<Interpreter> interpreters : interpreterGroup.values()) {
>> ....
>> for (Interpreter intp : interpreters) {
>> if (intp.getClassName().equals(className) || intp.getClass().getSuperclass().getName().equals(className)) {
>> interpreterFound = intp;
>> }
>>
>> ...
>> }
>>
>> ...
>> }
>> }
>> return null;
>> }
>>
>> Either of the two solutions would involve the modification of apache zeppelin code; do you think the change could be contributed to the community?, or maybe do you realize some other approach to change the way in which sub-interpreters of spark get the instance of spark interpreter?
>>
>> Any information about it I'll be attempt.
>>
>> Greetings
>>
>>
>> Jhon
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Jhon,
Do you mind to share what kind of custom function you want to add to spark
interpreter ? One idea in my mind is that we could add extension point to
the existing SparkInterpreter, and user can enhance SparkInterpreter via
these extension point. That means we just open some interfaces and users
can implement those interfaces, and just add their jars to spark
interpreter folder.
Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
> Hello!
>
> I'm a software developer and as part of a project I require to extend the
> functionality of SparkInterpreter without modifying it. I need instead
> create a new interpreter that extends it or wrap its functionality.
>
> I also need the spark sub-interpreters to use my new custom interpreter,
> but the problem comes here, because the spark sub-interpreters has a
> direct dependency to spark interpreter as they use the class name of spark
> interpreter to obtain its instance:
>
>
> private SparkInterpreter getSparkInterpreter() {
>
> ...
>
> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>
> }
>
>
> *Approach without modify apache zeppelin*
>
> My current approach to solve is to create a SparkCustomInterpreter that
> override the getClassName method as follows:
>
> public class SparkCustomInterpreter extends SparkInterpreter {
> ...
>
> @Override
> public String getClassName() {
> return SparkInterpreter.class.getName();
> }
> }
>
>
> and put the new class name in the interpreter-setting.json file of spark:
>
> [
> {
> "group": "spark",
> "name": "spark",
> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
> ...
> "properties": {...}
> }, ...
> ]
>
>
> The problem with this approach is that when I run a paragraph it fails. In
> general it fails because zeppelin uses both the class name of the
> instance and the getClassName() method to access the instance, and that
> causes many problems.
>
> *Approaches modifying apache zeppelin*
>
> There are two possible solutions related with the way in which the
> sub-interpreters get the SparkInterpreter instance class, one is getting
> the class name from a property:
>
>
> private SparkInterpreter getSparkInterpreter() {
>
> ...
>
> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>
> }
>
> And the other possibility is to modify the method
> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
> return the instance that whether has the same class name specified in the
> parameter or which super class has the same class name specified in the
> parameter:
>
>
> @ZeppelinApi
> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
> synchronized (interpreterGroup) {
> for (List<Interpreter> interpreters : interpreterGroup.values()) {
> ....
> for (Interpreter intp : interpreters) {
> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
> interpreterFound = intp;
> }
>
> ...
> }
>
> ...
> }
> }
> return null;
> }
>
>
> Either of the two solutions would involve the modification of apache
> zeppelin code; do you think the change could be contributed to the
> community?, or maybe do you realize some other approach to change the way
> in which sub-interpreters of spark get the instance of spark interpreter?
>
> Any information about it I'll be attempt.
>
> Greetings
>
>
> Jhon
>
Re: Extending SparkInterpreter functionality
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Jhon,
Do you mind to share what kind of custom function you want to add to spark
interpreter ? One idea in my mind is that we could add extension point to
the existing SparkInterpreter, and user can enhance SparkInterpreter via
these extension point. That means we just open some interfaces and users
can implement those interfaces, and just add their jars to spark
interpreter folder.
Jhon Anderson Cardenas Diaz <jh...@gmail.com>于2018年2月2日周五 上午5:30写道：
> Hello!
>
> I'm a software developer and as part of a project I require to extend the
> functionality of SparkInterpreter without modifying it. I need instead
> create a new interpreter that extends it or wrap its functionality.
>
> I also need the spark sub-interpreters to use my new custom interpreter,
> but the problem comes here, because the spark sub-interpreters has a
> direct dependency to spark interpreter as they use the class name of spark
> interpreter to obtain its instance:
>
>
> private SparkInterpreter getSparkInterpreter() {
>
> ...
>
> Interpreter p = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class.getName());
>
> }
>
>
> *Approach without modify apache zeppelin*
>
> My current approach to solve is to create a SparkCustomInterpreter that
> override the getClassName method as follows:
>
> public class SparkCustomInterpreter extends SparkInterpreter {
> ...
>
> @Override
> public String getClassName() {
> return SparkInterpreter.class.getName();
> }
> }
>
>
> and put the new class name in the interpreter-setting.json file of spark:
>
> [
> {
> "group": "spark",
> "name": "spark",
> "className": "org.apache.zeppelin.spark.SparkCustomInterpreter",
> ...
> "properties": {...}
> }, ...
> ]
>
>
> The problem with this approach is that when I run a paragraph it fails. In
> general it fails because zeppelin uses both the class name of the
> instance and the getClassName() method to access the instance, and that
> causes many problems.
>
> *Approaches modifying apache zeppelin*
>
> There are two possible solutions related with the way in which the
> sub-interpreters get the SparkInterpreter instance class, one is getting
> the class name from a property:
>
>
> private SparkInterpreter getSparkInterpreter() {
>
> ...
>
> Interpreter p = getInterpreterInTheSameSessionByClassName(*property.getProperty("zeppelin.spark.mainClass", SparkInterpreter.class.getName())* );
>
> }
>
> And the other possibility is to modify the method
> Interpreter.getInterpreterInTheSameSessionByClassName(String) in order to
> return the instance that whether has the same class name specified in the
> parameter or which super class has the same class name specified in the
> parameter:
>
>
> @ZeppelinApi
> public Interpreter getInterpreterInTheSameSessionByClassName(String className) {
> synchronized (interpreterGroup) {
> for (List<Interpreter> interpreters : interpreterGroup.values()) {
> ....
> for (Interpreter intp : interpreters) {
> if (intp.getClassName().equals(className) *|| intp.getClass().getSuperclass().getName().equals(className)*) {
> interpreterFound = intp;
> }
>
> ...
> }
>
> ...
> }
> }
> return null;
> }
>
>
> Either of the two solutions would involve the modification of apache
> zeppelin code; do you think the change could be contributed to the
> community?, or maybe do you realize some other approach to change the way
> in which sub-interpreters of spark get the instance of spark interpreter?
>
> Any information about it I'll be attempt.
>
> Greetings
>
>
> Jhon
>