You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by cloverhearts <gi...@git.apache.org> on 2017/09/08 07:07:29 UTC
[GitHub] zeppelin pull request #2574: [ZEPPELIN-2921 : 0.7x] does not work conda envi...
GitHub user cloverhearts opened a pull request:
https://github.com/apache/zeppelin/pull/2574
[ZEPPELIN-2921 : 0.7x] does not work conda environment in python interpreter
### What is this PR for?
It seems that the environment of the python interpreter has changed to the py4j environment, causing problems with the library path and the default environment.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2921
### How should this be tested?
Please run the following command line for each paragraph.
If the module such as scipy is normally imported, it is a success.
`%python.conda create --name Hello6 python=2.7`
`%python.conda activate Hello6`
`%python.conda install seaborn pandas numpy scipy matplotlib`
```
%python
import scipy as sp
import seaborn as sns
```
`%python.conda deactivate`
### Screenshots (if appropriate)
#### Before
#### After
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloverhearts/zeppelin fix/conda-interpreter-invalid-path-0.7
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2574.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2574
----
commit b287825358ff05347d9062d352485e5cf02d23d7
Author: CloverHearts <cl...@gmail.com>
Date: 2017-09-08T06:23:11Z
add conda local env
----
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts I have few questions.
What happens when we want to implement `PythonPyenvInterpreter` which is similar to `PythonCondaInterpreter`?
- In that case, we have to add new functions to PythonInterpreter like `setPyEnvLib()...`?
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts There is no problem about python lib paths.
`%python.conda install` command is the problem.
So you don't need to write code for python path.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
I think we are worried about `conda interpreter` and coupling for` conda env name`.
However, the conda env name is more important to the python interpreter than the conda interpreter.
Also, the python interpreter should not query the conda interpreter for its own state.
I think this misunderstanding is due to the member name `conda env name`.
If I have any additional problems, I will change this to `virtualEnvName`.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
https://travis-ci.org/malayhm/zeppelin/jobs/272788659
https://travis-ci.org/1ambda/zeppelin/jobs/273195461
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@1ambda I'll check it, thank you :)
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
In my opinion, allowing for fluctuations in the python environment is itself a problem.
But zeppelin are supporting it.
If it does not, then the semantics of conda activate and conda deactivate disappear.
In fact, I also think that changing the python environment can be a problem.
However, the current Zeppelin supports it, and a bug exists.
This PR is a PR that solves the bug.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
Thank you @1ambda @zjffdu @felixcheung
I think my code with various purposes is distracting you.
And, unfortunately, my English ability is difficult to describe in a sophisticated way.
Therefore, I will share PR.
In this PR, we will only deal with the wrong library installation of conda.
Many thanks for your comments.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@zjffdu @1ambda @felixcheung
Thank you for your kind reply despite my lack of English.
I hope this PR is merged.
So, i will try to reflect your opinions as much as possible.
Thank you. :)
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
I have a solution to remove conda related members from PythonInterpreter.
also PythonCondaInterpreter will not have any members.
However, it differs in focus from this PR.
(The problem is that there are changes related to the start of the Python Interpreter.)
This PR is intended to be used when the user changes the environment by installing new python while using conda.
This also solves the Zeppelin Python Interpreter's infinite loop problem.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@zjffdu @felixcheung
Thank you for your opinions :)
I was left my comment.
Could you check this one?
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts
`There is only one python library path to be added.`
Do you mean the py4j ? IIUC, PythonInterpreter will add py4j to PYTHONPATH, it doesn't relate with `PythonCondaInterpreter`. That's why I mention we just need to set `zeppelin.python`, Let me know if I miss anything.
---
[GitHub] zeppelin pull request #2574: [ZEPPELIN-2921 : 0.7x] does not work conda envi...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts closed the pull request at:
https://github.com/apache/zeppelin/pull/2574
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@1ambda Thank you, and i will more test :)
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts There is no problem about python lib paths.
`%python.conda install` and `%python conda list` commands are the problem. They don't aware activated env by Zeppelin.
So you don't need to write code for python path.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts Just to clarify my concern. Here's what I see when I use conda outside of zeppelin.
Step 1. Create enviroment `env_1` using conda, and install pandas
Step 2. Set `zeppelin.python` to the binPath of `env_1` and restart python interpreter
Step 3. Now when I use python interpreter, I am using python of `env_1` and can use any libraries installed under `env_1`
That means I just need to set `zeppelin.python` to switch to another conda env. That's why I don't understand why you need to set additional Python library path for the new environment
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@1ambda
Currently there is a possibility that it will be structurally.
However, if you store it in PythonCondaInterpreter, the problem is greatly aggravated.
1. PythonInterpreter needs to monitor PythonCondaInterpreter at all times.
(Python Interpreter should check Conda's environment activation when restarting.)
2. Other configurations other than PythonCondaInterpreter If an interpreter is added (as you might have guessed), the Python Interpreter is in a situation where you need to constantly query for all other Interpreter environments. This will affect the life cycle of the Python Interpreter.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@zjffdu
Thank you for your opinion,
I present scenarios for environment changes using conda.
Python version and module names are arbitrary, so do not worry.
What is important is change in every environment.
1. Basically Zeppelin's python
python: 2.71
installed library: (none)
use to zeppelin
```
%python
import sys
print(sys.version)
print(sys.path)
```
result
```
python version 2.7.1
%ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME
```
```
%python
import myModule
```
result
```
no module myModule
```
2. install conda and new python
```
%python.conda create --name python3 python=3.4
```
```
%python.conda activate python3
```
```
%python
import sys
print(sys.version)
print(sys.path)
```
result
```
python version 3.4
%ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME:%CONDA_ENV%/python3/lib
```
```
%python
import myModule
```
result
```
no module myModule
```
try again
```
%python.conda install myModule
```
```
import myModule
```
result
```
imported.
```
3. restore python environment
```
%python.conda deactivate
```
check for recovered to the original Zeppelin environment.
```
%python
import sys
print(sys.version)
print(sys.path)
```
result
```
python version 2.7.1
%ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME
```
```
import myModule
```
result
```
no module myModule
```
All three must be supported by conda.
1. Support to installing and using the python library
2. Support to change the python version
3. Support to configuring non-python environments
old zeppelin does works for this part(0.71).
but, the Python Interpreter has changed significantly since 0.71.
I guess missing the environment part for this.
I modified it.
python env set code
(https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java#L151)
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
```
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.456 sec <<< FAILURE! - in org.apache.zeppelin.interpreter.remote.RemoteInterpreterServerTest
testStartStop(org.apache.zeppelin.interpreter.remote.RemoteInterpreterServerTest) Time elapsed: 10.037 sec <<< FAILURE!
java.lang.AssertionError: expected:<true> but was:<false>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServerTest.testStartStop(RemoteInterpreterServerTest.java:62)
```
This error occurs equally in other PRs and branches.
It has been confirmed that it is not related to my changes.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@cloverhearts I am getting this error.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/2574
Thanks for the fix. Let me test and feedback soon.
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
I have separated this PR.
I'm sorry for the confusion.
https://github.com/apache/zeppelin/pull/2578
---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/2574
@zjffdu
This change will work exactly as your request.
There is only one python library path to be added.
1. If the user does not install python via conda.
(If the user installs only the library)
Zeppelin use to `zeppelin.python`.
(Used in the original environment.)
2. If you install python via conda.
(separate python from zeppelin.python)
if when the conda environment is active, add to library path and replace to installed python via conda from zeppelin.python.
Zeppelin to work in the new python environment.
3. The user disables the environment. (conda deactiavte)
It is restored to the original zeppelin.python environment.
---