You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Leemoonsoo <gi...@git.apache.org> on 2017/01/27 03:40:40 UTC
[GitHub] zeppelin pull request #1952: [ZEPPELIN-753] TableData abstraction
GitHub user Leemoonsoo opened a pull request:
https://github.com/apache/zeppelin/pull/1952
[ZEPPELIN-753] TableData abstraction
### What is this PR for?
This PR implements TableData abstraction that every interpreter can use to share data through resource pool.
This PR is made on top of #1951. Once #1951 is merged, this PR will be rebased to master.
Meanwhile, to review,
TableData.java - Definition of TableData abstraction
TableDataProxy.java - Proxy object of TableData. This enables data access through ResourcePool
InterpreterResultTableData - Convert Table type interpret result into TableData. Each interpreter can implement it's own TableData abstraction. For example JDBCTableData, DataFrameTableData, and so on.
### What type of PR is it?
Feature
### Todos
* [x] - TableData
* [x] - TableDataProxy
* [x] - Unittest
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-753
### How should this be tested?
Unittest included
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/zeppelin ZEPPELIN-753
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1952.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1952
----
commit 74b502e9c89aebe09c932f0798817a1c5398bead
Author: Lee moon soo <mo...@apache.org>
Date: 2017-01-26T00:39:42Z
Support method invocation on the resource in local,remote resource pool
commit 30e62f23eebfe8ca441f3c36c00605132b3779cf
Author: Lee moon soo <mo...@apache.org>
Date: 2017-01-26T02:12:03Z
make RemoteResource.invokeMethod returns Resource when return resource name is specified
commit 75a9b32357eb911d923d76a1c331e17d9d4ad4ce
Author: Lee moon soo <mo...@apache.org>
Date: 2017-01-25T17:41:29Z
TableData abstraction
commit 1751ee015b5bdb936e45f442c5bb9e2ec2fc52b2
Author: Lee moon soo <mo...@apache.org>
Date: 2017-01-26T02:38:45Z
Proxy for tabledata
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1952: [ZEPPELIN-753] TableData abstraction
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1952
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1952: [ZEPPELIN-753] TableData abstraction
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1952
@Leemoonsoo Nice to have abstracted table structure inside zeppelin. It enable us many things even i just glanced at.
I will take look deeply and comment soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1952: [ZEPPELIN-753] TableData abstraction
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1952
@1ambda Thanks for taking a look.
Regarding query directly over the TableData in the ResourcePool, it depends on each interpreter.
For example, SparkInterpreter can implement [DataSource API](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala) to load data from ResourcePool. Then SparkInterpreter will able to query data created from other interpreter, when the data is abstracted by TableData and exist in the ResourcePool.
The same approaches can be applied to all other interpreter. So any interpreter that supports plugin data source layer, might implement ResourcePool as a data source and make query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1952: [ZEPPELIN-753] TableData abstraction
Posted by 1ambda <gi...@git.apache.org>.
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1952
Sorry for late reply. I was taking a look also recent resource related commit to understand fully.
1. This PR is well abstracted and shipped with enough unit test.
2. And easy to share table type interpreter results with other interpreters. The only thing interpreter develop should do is creating it's own table data class (e.g `JDBCTableData`, `DataFrameTableData` as Leemoonsoo mentioned.) to be inserted into pool using interpreter specific low-level result (e.g `JDBCResultSet`, ...)
3. This also enable us to create table data view for shared result in pools.
@Leemoonsoo Now i have a question.
- I can't imagine how one interpreter (e.g JDBC) is able to use shared table result from another interpreter (e.g spark.sql). I know we have APIs to get results from pool. But getting doesn't mean using (e.g querying) directly.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1952: [ZEPPELIN-753] TableData abstraction
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1952
@1ambda further discussion about "how interpreter access data from other interpreter" can be done in https://issues.apache.org/jira/browse/ZEPPELIN-2019.
Merge this PR to master if no further discussions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1952: [ZEPPELIN-753] TableData abstraction
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1952
@1ambda do you have any comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---