You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by hammertank <gi...@git.apache.org> on 2017/04/21 03:35:45 UTC
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
GitHub user hammertank opened a pull request:
https://github.com/apache/zeppelin/pull/2271
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
### What is this PR for?
Fix memory leak under scoped mode of SparkInterpreter which is caused by inapproprately setting Thread.contextClassLoader.
### What type of PR is it?
Bug Fix
### Todos
Rest Thread.contextClassLoader in Methods:
* [ ] - public void open()
* [ ] - private Results.Result interpret(String line)
* [ ] - public List<InterpreterCompletion> completion(String buf, int cursor)
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2414
### How should this be tested?
TestCase SparkInterpreterTest#testContextClassLoader
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hammertank/zeppelin ZEPPELIN-2414
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2271
----
commit 312b972bfd1a33cf1ec439e480795be151f63c87
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-18T02:42:23Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit 8be737b120c1a614547df17454c03da2848aefe6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T02:32:27Z
[HOTFIX] CAN'T install cran packages using apt in Travis CI
commit 79d005fd64b4e9e9334975611d6dd421b0258e30
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:12:23Z
[ZEPPELIN-1455] Fix flaky test: AbstractAngularElemTest
commit a7953cffd10f89ead29987281fd9052e8c5cc6b9
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:54:24Z
Fix CI build failure on branch-0.6
commit e971e610d2e626f310f123efd2ef09c26f432a85
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T06:58:25Z
[ZEPPELIN-1972] Preserve context classloader
commit 36c82419b27646df8bc0b6f6aac224b23f7ee630
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T07:51:36Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 9df575da5d6998135e010c44f2b296850536ee49
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:20:30Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit d59ea0836a03c7e55228ff2888d0261b8222c7c2
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:21:42Z
Add travis_check.py from master
commit 959011130c2dfa2cccdb0b68c8d9f12020383d18
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T10:19:10Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 230e5db11f101ce99127f3a2fc1972e3a0bcddb6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T05:28:11Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit f5b6f1238e258f2ea7a33ba524b19bea243acfd7
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T06:26:37Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2271
Thanks @hammertank for the contribution.
What do you think wrap interpreter using [ClassLoaderInterpreter](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/ClassloaderInterpreter.java) instead of modifying individual interpreter?
Interpreter is being instantiated from [RemoteInterpreterServer](https://github.com/apache/zeppelin/blob/branch-0.6/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java#L166), so this is place where ClassLoaderInterpreter can wrap Interpreter. Like [Interpreter factory does](https://github.com/apache/zeppelin/blob/branch-0.6/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java#L894)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank closed the pull request at:
https://github.com/apache/zeppelin/pull/2271
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
GitHub user hammertank reopened a pull request:
https://github.com/apache/zeppelin/pull/2271
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
### What is this PR for?
Fix memory leak under scoped mode of SparkInterpreter which is caused by inapproprately setting Thread.contextClassLoader.
### What type of PR is it?
Bug Fix
### Todos
Rest Thread.contextClassLoader in Methods:
* [ ] - public void open()
* [ ] - private Results.Result interpret(String line)
* [ ] - public List<InterpreterCompletion> completion(String buf, int cursor)
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2414
### How should this be tested?
TestCase SparkInterpreterTest#testContextClassLoader
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hammertank/zeppelin ZEPPELIN-2414
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2271
----
commit 312b972bfd1a33cf1ec439e480795be151f63c87
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-18T02:42:23Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit 8be737b120c1a614547df17454c03da2848aefe6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T02:32:27Z
[HOTFIX] CAN'T install cran packages using apt in Travis CI
commit 79d005fd64b4e9e9334975611d6dd421b0258e30
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:12:23Z
[ZEPPELIN-1455] Fix flaky test: AbstractAngularElemTest
commit a7953cffd10f89ead29987281fd9052e8c5cc6b9
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:54:24Z
Fix CI build failure on branch-0.6
commit e971e610d2e626f310f123efd2ef09c26f432a85
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T06:58:25Z
[ZEPPELIN-1972] Preserve context classloader
commit 36c82419b27646df8bc0b6f6aac224b23f7ee630
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T07:51:36Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 9df575da5d6998135e010c44f2b296850536ee49
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:20:30Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit d59ea0836a03c7e55228ff2888d0261b8222c7c2
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:21:42Z
Add travis_check.py from master
commit 959011130c2dfa2cccdb0b68c8d9f12020383d18
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T10:19:10Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 230e5db11f101ce99127f3a2fc1972e3a0bcddb6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T05:28:11Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit f5b6f1238e258f2ea7a33ba524b19bea243acfd7
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T06:26:37Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit a43965d52dd4e11d4c74e0d5fa6afbde1d3f9f5e
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-21T08:24:49Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
GitHub user hammertank reopened a pull request:
https://github.com/apache/zeppelin/pull/2271
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
### What is this PR for?
Fix memory leak under scoped mode of SparkInterpreter which is caused by inapproprately setting Thread.contextClassLoader.
### What type of PR is it?
Bug Fix
### Todos
Rest Thread.contextClassLoader in Methods:
* [x] - Wraps interpreters with ClassLoader with ClassLoaderInterpreter in RemoteInterpreterServer.createInterpreter
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2414
### How should this be tested?
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hammertank/zeppelin ZEPPELIN-2414
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2271
----
commit 2b67899ffe71658ca99f802005e1c5e20ffa0db7
Author: z0621 <zh...@dtdream.com>
Date: 2017-05-04T05:57:30Z
Wraps interpreters with ClassloaderInterpreters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2271
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
GitHub user hammertank reopened a pull request:
https://github.com/apache/zeppelin/pull/2271
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
### What is this PR for?
Fix memory leak under scoped mode of SparkInterpreter which is caused by inapproprately setting Thread.contextClassLoader.
### What type of PR is it?
Bug Fix
### Todos
Rest Thread.contextClassLoader in Methods:
* [ ] - public void open()
* [ ] - private Results.Result interpret(String line)
* [ ] - public List<InterpreterCompletion> completion(String buf, int cursor)
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2414
### How should this be tested?
TestCase SparkInterpreterTest#testContextClassLoader
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hammertank/zeppelin ZEPPELIN-2414
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2271
----
commit 312b972bfd1a33cf1ec439e480795be151f63c87
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-18T02:42:23Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit 8be737b120c1a614547df17454c03da2848aefe6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T02:32:27Z
[HOTFIX] CAN'T install cran packages using apt in Travis CI
commit 79d005fd64b4e9e9334975611d6dd421b0258e30
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:12:23Z
[ZEPPELIN-1455] Fix flaky test: AbstractAngularElemTest
commit a7953cffd10f89ead29987281fd9052e8c5cc6b9
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T03:54:24Z
Fix CI build failure on branch-0.6
commit e971e610d2e626f310f123efd2ef09c26f432a85
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T06:58:25Z
[ZEPPELIN-1972] Preserve context classloader
commit 36c82419b27646df8bc0b6f6aac224b23f7ee630
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T07:51:36Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 9df575da5d6998135e010c44f2b296850536ee49
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:20:30Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit d59ea0836a03c7e55228ff2888d0261b8222c7c2
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T09:21:42Z
Add travis_check.py from master
commit 959011130c2dfa2cccdb0b68c8d9f12020383d18
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-19T10:19:10Z
Merge branch 'HOTFIX' into ZEPPELIN-2414
commit 230e5db11f101ce99127f3a2fc1972e3a0bcddb6
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T05:28:11Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit f5b6f1238e258f2ea7a33ba524b19bea243acfd7
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-20T06:26:37Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit a43965d52dd4e11d4c74e0d5fa6afbde1d3f9f5e
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-21T08:24:49Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
commit 87094a420f3e34485f9169b3853526e596767e93
Author: z0621 <zh...@dtdream.com>
Date: 2017-04-21T09:54:12Z
[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by inapproprately setting Thread.contextClassLoader
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank commented on the issue:
https://github.com/apache/zeppelin/pull/2271
@Leemoonsoo Changed the code as you suggested. But I have no idea how to write a testcase to validate the modification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank commented on the issue:
https://github.com/apache/zeppelin/pull/2271
Found "ClassNotFoundException" after applying this fix. This is caused by SparkContext using the first TranslatingClassLoader to deserialize TaskResult.
Sorry to say that I haven't figure out a solution yet. :stuck_out_tongue_closed_eyes:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank closed the pull request at:
https://github.com/apache/zeppelin/pull/2271
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2271
You can close/reopen the issue to trigger Jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank commented on the issue:
https://github.com/apache/zeppelin/pull/2271
@Leemoonsoo re-commit as required. Please review again.
Jenkins did not get the final travis-ci results successfully. Is there a way to trigger Jenkins to fetch again without a push?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2271
#2262 has been merged. so now, could you try include commit only related to this contributions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2271: [ZEPPELIN-2414] Memory leak under scoped mode of Spark...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank commented on the issue:
https://github.com/apache/zeppelin/pull/2271
@Jongyoul Lee Ready to review. Please help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...
Posted by hammertank <gi...@git.apache.org>.
Github user hammertank closed the pull request at:
https://github.com/apache/zeppelin/pull/2271
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---