You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by haohui <gi...@git.apache.org> on 2015/12/03 00:18:45 UTC
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
GitHub user haohui opened a pull request:
https://github.com/apache/storm/pull/914
STORM-1040. SQL support for Storm.
This PR rebases all the changes in the STORM-1040 branch towards the latest master.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/haohui/storm STORM-1040-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/914.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #914
----
commit 9729999e29f5543df3ac3dc5dbe798ee59eafc1f
Author: Haohui Mai <wh...@apache.org>
Date: 2015-09-15T16:30:50Z
STORM-1045. Parse CREATE TABLE statements.
commit 0d20de18f9a52d6006e7c1a18ee586143a078264
Author: Haohui Mai <wh...@apache.org>
Date: 2015-09-22T23:12:12Z
STORM-1060. Serialize Calcite plans into JSON format.
commit ed68bcca602b7e48a3f7fd6d5cf70e45b7fb95f9
Author: Haohui Mai <wh...@apache.org>
Date: 2015-10-22T23:42:24Z
Implement compiler for expressions.
commit 94c471d7fa75c2711da0a8e1d2b6332e95b88fc3
Author: Haohui Mai <wh...@apache.org>
Date: 2015-10-22T23:45:20Z
Implement compiler for relation nodes.
commit cb2c328556e6fc9c1e550219706bfe3d6368ef3c
Author: Haohui Mai <wh...@apache.org>
Date: 2015-10-22T23:52:59Z
Compile logical plans to Java code.
commit 1f3d6fc5caac612ff2327c2ab9e6bfd2243f3363
Author: Haohui Mai <wh...@apache.org>
Date: 2015-10-27T00:29:20Z
STORM-1130. Support AND, OR, NOT operators in StormSQL.
commit b31a91ff564a02f30e28e54be5e1a178113bf8be
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-02T22:34:20Z
Implement push-based APIs for StormSQL.
commit 4e76a4221aefa23f8afeec5b34c38c312d0062b6
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-02T23:35:29Z
STORM-1149. Support pluggable data sources in CREATE TABLE.
commit 8fb17918bb2937b7ff99447b86f67b5215e615c8
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-03T23:06:22Z
Implement IS operators in StormSQL.
commit d17b685eb15e0858acbf471712fdd7620517f092
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-04T17:24:32Z
Implement nullable semantics for AND, OR and NOT operators.
commit 883fca19ca4ff66f98864aa891940e43e3149982
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-04T23:51:11Z
STORM-1173. Support string operations in StormSQL.
commit 804aed25c495abc50581375735c591ce04a456b2
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-05T02:36:59Z
STORM-1159. Support nullable operations in StormSQL.
commit 7afc697ddacf63b85f3b65910868603a5769984b
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-06T23:50:31Z
Refactor to support compiling StormSQL to Trident topology.
commit b7ccef722e57fd43c188213a5a49f5500a44fb5f
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-11T18:03:07Z
Allow deserializing Java class with custom classloaders in tests.
commit 602a15bef9ed5771dc9fe684d995aad6617ea89c
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-11T22:23:14Z
STORM-1200. Support collations of primary keys.
commit 2b9cd6536dc613f775136f6185cb450c335a020c
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-16T22:48:43Z
STORM-1181. Compile SQLs into Tridient topology and execute them in LocalCluster.
commit 4268327759124cb2861f8c71b94b0b749ad01cfe
Author: Haohui Mai <wh...@apache.org>
Date: 2015-11-19T22:09:26Z
STORM-1222. Support Kafka as external tables in StormSQL.
commit cc57c155f90e8e888b2f37bd5e77c8345f85f333
Author: Haohui Mai <wh...@apache.org>
Date: 2015-12-01T02:58:29Z
STORM-1357. Support writing to Kafka streams in Storm SQL.
commit 3f512370eeefaa0bfe2cb3a6a83cb30580c4d02a
Author: Haohui Mai <wh...@apache.org>
Date: 2015-12-02T09:02:28Z
STORM-1201. Support distributed deployment in StormSQL.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by darionyaphet <gi...@git.apache.org>.
Github user darionyaphet commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161654555
@haohui Thank you for you Great Work :+1: I want to know which SQL pattern had support ?
such as count , distinct and sum ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161486071
+1. Great work @haohui to pull it off :). Looking forward to get this into master branch. I've ran the tests with java 7 and java 8 all the tests are passing . Travis build failures not relevant to this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161738187
For some reasons I can't reopen this PR now. Created #919 as a duplicate for this PR.
Thanks everybody for the comments. I have addressed them in the latest PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161491178
+1 Great work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/914#discussion_r46582960
--- Diff: external/sql/README.md ---
@@ -0,0 +1,98 @@
+# Storm SQL
+
+Compile SQL queries to Storm topologies.
+
+## Usage
+
+Run the ``storm sql`` command to compile SQL statements into Trident topology, and submit it to the Storm cluster
+
+```
+$ bin/storm sql <topo-name> <sql-file>
+```
+
+In which `topo-name` is the name of the topology, and `sql-file` contains a list of SQL statements to be executed.
+
+## Supported Features
+
+The following features are supported in the current repository:
+
+* Streaming from and to external data sources
+* Filtering tuples
+* Projections
+
+Aggregation, windowing and joining tables are yet to be implemented. Specifying parallelism hints is not yet supported.
+
+## Specifying External Data Sources
+
+In StormSQL data is represented by external tables. Users can specify data sources using the `CREATE EXTERNAL TABLE` statement. For example, the following statement specifies a Kafka spouts and sink:
+
+```
+CREATE EXTERNAL TABLE FOO (ID INT PRIMARY KEY) LOCATION 'kafka://localhost:2181/brokers?topic=test' TBLPROPERTIES '{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"storm.kafka.IntSerializer","value.serializer":"storm.kafka.ByteBufferSerializer"}}'
+```
+
+The syntax of `CREATE EXTERNAL TABLE` closely follows the one defined in [Hive Data Definition Language](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL).
+
+## Plugging in External Data Sources
+
+Users plug in external data sources through implementing the `ISqlTridentDataSource` interface and registers them using the mechanisms of Java's service loader. The external data source will be chosen based on the scheme of the URI of the tables. Please refer to the implementation of `storm-sql-kafka` for more details.
+
+It is also important to note that users also need to provide the dependency of the external data sources in the `extlib` directory. Otherwise the topology will fail to run because of `ClassNotFoundException`.
--- End diff --
can you highlight this part as a limitation section and file relevant JIRAs and include them in the README as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by haohui <gi...@git.apache.org>.
Github user haohui closed the pull request at:
https://github.com/apache/storm/pull/914
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161738550
Currently the implementation does not support any aggregation (i.e., count, sum, max) yet. It will be added once StormSQL supports windowing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1040. SQL support for Storm.
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the pull request:
https://github.com/apache/storm/pull/914#issuecomment-161718691
+1. Nice job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---