You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by tinkoff-dwh <gi...@git.apache.org> on 2017/03/05 09:12:09 UTC
[GitHub] zeppelin pull request #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
GitHub user tinkoff-dwh opened a pull request:
https://github.com/apache/zeppelin/pull/2096
[ZEPPELIN-2194] precode for PySparkInterpreter
### What is this PR for?
Added parameter `zeppelin.pyspark.precode` to PySparkInterpreter. This is snippet of code which executes when interpreter initialize.
### What type of PR is it?
Feature
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2194
### How should this be tested?
1. Set parameter **zeppelin.pyspark.precode** `precodeVar='some text'`
2. Run
```
%pyspark
print(precodeVar)
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tinkoff-dwh/zeppelin ZEPPELIN-2194
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2096.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2096
----
commit 357ac2368a5d62597a0dbc9cd60ed83a5ec428a4
Author: Tinkoff DWH <ti...@gmail.com>
Date: 2017-03-03T08:06:28Z
[ZEPPELIN-2194] precode for PySparkInterpreter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by masyan <gi...@git.apache.org>.
Github user masyan commented on the issue:
https://github.com/apache/zeppelin/pull/2096
close works, but open not. I don't understad why. Maybe run in
if (!lazy.isOpen()) {
lazy.open();
interpreter.interpret(interpetator.getProperty(propertyName), context);
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by tinkoff-dwh <gi...@git.apache.org>.
Github user tinkoff-dwh closed the pull request at:
https://github.com/apache/zeppelin/pull/2096
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by masyan <gi...@git.apache.org>.
Github user masyan commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@Leemoonsoo
so not working as I expected, method open not runs (in RemoteInterpreterServer). need to better understand the circuit how it works
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by masyan <gi...@git.apache.org>.
Github user masyan commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@Leemoonsoo
Although probably implement hooks to do this properly. This may confuse the user. If he is going to register these hooks in their paragraphs, and they will not work (because they will be executed only when the initialization of the interpreter).
I propose, then, is a global solution. Add two parameters
zeppelin.interpretator.after_open
zeppelin.interpretator.before_close
and add `interpetator.interpret(interpetator.getProperty(propertyName))` to methods open and close (RemoteInterpreterServer.java)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by tinkoff-dwh <gi...@git.apache.org>.
Github user tinkoff-dwh commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@Leemoonsoo
https://issues.apache.org/jira/browse/ZEPPELIN-2216
what to do with this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2096
Also, is `precode` a concept at the interpreter instance level?
Would there be need to run different code per note - how is this going to interact with interpreter mode like `per note`, `isolated` and so on?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by tinkoff-dwh <gi...@git.apache.org>.
Github user tinkoff-dwh commented on the issue:
https://github.com/apache/zeppelin/pull/2096
#2221 general solution
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@masyan
Do you mind open PR or JIRA issue and discuss there for more generic way of run code before/after open/close ? I think it can be little complicated to discuss here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@masyan Make sense!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2096
Thanks @tinkoff-dwh for the contribution.
I've seen #2078.
If `precode` is going to be added to all interpreter, how about create some facility that runs code when initialize interpreter and then make feature based on it?
How about extend [Interpreter.Hook()](https://github.com/apache/zeppelin/blob/branch-0.7/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java#L208) which currently has [PRE_EXEC](https://github.com/apache/zeppelin/blob/branch-0.7/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterHookRegistry.java#L143) and [POST_EXEC](https://github.com/apache/zeppelin/blob/branch-0.7/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterHookRegistry.java#L146) as event type that make code run before and after the interpret() call respectively.
I think we can add event type such as `AFTER_OPEN`, `BEFORE_CLOSE`. Then with minimum modification of each interpreter, we can have the feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by masyan <gi...@git.apache.org>.
Github user masyan commented on the issue:
https://github.com/apache/zeppelin/pull/2096
I think this is a good idea. I'm trying to implement this and test task [#2085](https://github.com/apache/zeppelin/pull/2085)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by tinkoff-dwh <gi...@git.apache.org>.
Github user tinkoff-dwh commented on the issue:
https://github.com/apache/zeppelin/pull/2096
Ready to review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by masyan <gi...@git.apache.org>.
Github user masyan commented on the issue:
https://github.com/apache/zeppelin/pull/2096
and InterpreterContext is unavailable in method close
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2096: [ZEPPELIN-2194] precode for PySparkInterpreter
Posted by tinkoff-dwh <gi...@git.apache.org>.
Github user tinkoff-dwh commented on the issue:
https://github.com/apache/zeppelin/pull/2096
@felixcheung
1. Yes.
2. Code will be executed after method open() (at the end of this method).
accordingly:
1) Per Note - executes once for note + Interpreter. For different code, percode is equals (the same).
2) Per User - executes once for each user + Interpreter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---