You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Leemoonsoo <gi...@git.apache.org> on 2016/01/19 04:45:26 UTC
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
GitHub user Leemoonsoo opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/655
[ZEPPELIN-619] Shared Resource pool across interpreter processes
### What is this PR for?
This is sub task of https://issues.apache.org/jira/browse/ZEPPELIN-533.
It provides shared resource pool to exchange data across interpreter processes.
### What type of PR is it?
Feature
### Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-619
### How should this be tested?
create two different spark interpreter settings.
create two different notebooks each bind different spark interpreter setting.
put an object from one notebook.
read the object from the other notebook. (from the other interpreter process)
See screenshot
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? javadoc included.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/incubator-zeppelin resource_pool
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/655.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #655
----
commit 4620d991361d763a1f080224cada27264324cf20
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-11T20:43:17Z
distributed resource pool
commit 85aadbbfead315212d5f3c0a473609be1ecbfd4a
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-18T20:30:17Z
distributed resource pool across interpreter processes
commit 5ba70272d1f8987f108c603570fc9b14fe3fffaf
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-18T20:46:50Z
Update test
commit d651f3e2aadf72434227d8bf84189338dd21d8c0
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-19T03:02:50Z
Fix test
commit fbf31bcfcce6829bc2c3e83b6e3d246bb0e21098
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-19T03:22:21Z
ZeppelinContext provides api for resource pool
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-177298867
@doanduyhai Thanks for review and i think i addressed all your comment. Please take a look.
Regarding ResourcePool and AngularObjectRegistry, here's differences
- ResourcePool is not propagating object from/to front-end, but AngularObjectRegistry do.
- ResourcePool is not persisted into note.json but AngularObjectRegistry does.
- AngularObjectRegistry has scope (paragraph/notebook) but ResourcePool does not.
I agree, if we improve ResourcePool more and replace AngularObjectRegistry, that would be simpler and better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by doanduyhai <gi...@git.apache.org>.
Github user doanduyhai commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-177315262
@Leemoonsoo Thanks for the detailed explanation of differences between AngularObjectRegistry and ResourcePool. I guess that the AngularObjectRegistry is still very relevant since it has a scope system and is saved with notebooks in json.
Otherwise, the code looks good to me so +1 for merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by doanduyhai <gi...@git.apache.org>.
Github user doanduyhai commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-172816425
A question @Leemoonsoo, it resource sharing only available through the ZeppelinContext object of the Spark interpreter or is it also available to other interpreters ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by doanduyhai <gi...@git.apache.org>.
Github user doanduyhai commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-176113214
Another question, now that we have a resource pull to share data between interpreters, should we deprecate AngularObjectRegistry and use resource pool instead ?
I have the feeling that resource pool is more general purpose. Furthermore with AngularObject we have the notion of scope (paragraph or note) which we don't have with resource pool. It helps segregate data properly
Anyway it is just a question, it's not blocking for this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo closed the pull request at:
https://github.com/apache/incubator-zeppelin/pull/655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-177322651
Thanks @doanduyhai for the review.
Merge if there're no more discussions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by Leemoonsoo <gi...@git.apache.org>.
GitHub user Leemoonsoo reopened a pull request:
https://github.com/apache/incubator-zeppelin/pull/655
[ZEPPELIN-619] Shared Resource pool across interpreter processes
### What is this PR for?
This is sub task of https://issues.apache.org/jira/browse/ZEPPELIN-533.
It provides shared resource pool to exchange data across interpreter processes.
### What type of PR is it?
Feature
### Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-619
### How should this be tested?
create two different spark interpreter settings.
create two different notebooks each bind different spark interpreter setting.
put an object from one notebook.
read the object from the other notebook. (from the other interpreter process)
See screenshot
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? javadoc included.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/incubator-zeppelin resource_pool
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/655.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #655
----
commit 020a4d150c838390bce523997dd1ba6794e2479d
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-11T20:43:17Z
distributed resource pool
commit 1eff13e4933de430608a8e03fea0c46a840341f7
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-18T20:30:17Z
distributed resource pool across interpreter processes
commit 8980d8da2602dc27e90b569d89aae8285ed21343
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-18T20:46:50Z
Update test
commit 269b961291c638cd42ac8ecd14e28e3bf1bfe6ff
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-19T03:02:50Z
Fix test
commit 9f1105de6545d4ffa49122cf61853b85a05d84d0
Author: Lee moon soo <mo...@apache.org>
Date: 2016-01-19T04:06:04Z
ZeppelinContext provides api for resource pool
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by doanduyhai <gi...@git.apache.org>.
Github user doanduyhai commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-172970950
Great news! We have finally a way to share variables between interpreters!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/655#issuecomment-172945821
@doanduyhai
ZeppelinContext expose convenient API for ResourcePool for scala repl.
ResourcePool is available through InterpreterContext.getResourcePool() and InterpreterContext is injected into every Interpreter.interpret() call.
So it is available to all other interpreters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: [ZEPPELIN-619] Shared Resource po...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-zeppelin/pull/655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---