You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by HeartSaVioR <gi...@git.apache.org> on 2016/08/23 07:18:41 UTC
[GitHub] storm pull request #1643: STORM-2050 [storm-sql] Support User Defined Aggreg...
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1643
STORM-2050 [storm-sql] Support User Defined Aggregate Function for Trident mode
NOTE: This pull request is on top of STORM-1434 (#1635)
After this patch, below sql statement runs fine for cluster mode (Trident).
(MyConcat is an user defined aggregate function.)
```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, NAME VARCHAR, UNIT_PRICE INT, QUANTITY INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES '{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE SUMMARY_ORDERS (ID INT PRIMARY KEY, ID_PLUS_3 INT, CNT INT, NAME_CONCAT VARCHAR) LOCATION 'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES '{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE FUNCTION MYPLUS AS 'org.apache.storm.sql.TestUtils$MyPlus'
CREATE FUNCTION MYCONCAT AS 'org.apache.storm.sql.TestUtils$MyConcat'
INSERT INTO SUMMARY_ORDERS SELECT ID, MYPLUS(ID, 3) AS ID_PLUS_3, COUNT(*) AS CNT, MYCONCAT(NAME) AS NAME_CONCAT FROM ORDERS GROUP BY ID
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-2050-on-top-of-STORM-1434
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1643.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1643
----
commit 779fea67ca68fde2a197ec98b2524312295b4737
Author: Jungtaek Lim <ka...@gmail.com>
Date: 2016-08-18T07:32:46Z
STORM-1434 Support the GROUP BY clause in StormSQL
* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is needed
*** Janino comes in to help evaluating code block in runtime
* Add test for scala UDF with Trident
commit 02ed2df07227042248c5ac2614d7f7692d79c529
Author: Jungtaek Lim <ka...@gmail.com>
Date: 2016-08-23T07:12:31Z
STORM-2050 [storm-sql] Support User Defined Aggregate Function for Trident mode
* support UDAF with single parameter for Trident mode
* doesn't support 'jar' option yet
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
Need to rebase again with current master (applied STORM-1434) but there're lots of conflict so would like to re-work based on current instead of resolving conflict.
Will leave a comment once it's done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
Addressed RAT issue, and rebased to latest pull request of STORM-1434 (#1635)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
For now, JAR option is not supported yet.
Btw, I'm thinking that JAR option is not needed after STORM-2016 is introduced. If user want to include additional jar for UDF / UDAF, user can just include that jar with `--jars` or `--artifacts` option.
It might be used for handling UDF / UDAF class with isolating classloader, but I'd love to make it simple, and address when someone claims it is needed.
I'd like to see @arunmahadevan 's opinion since he introduced DDL for UDF.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1643: STORM-2050 [storm-sql] Support User Defined Aggreg...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1643
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
Found RAT failing (1 file). Will check and address.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
OK. Rebased with current master.
@ptgoetz @harshach @haohui @arunmahadevan Please take a look. Thanks in advance!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1643
@arunmahadevan Thanks for reviewing!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1643: STORM-2050 [storm-sql] Support User Defined Aggregate Fun...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on the issue:
https://github.com/apache/storm/pull/1643
+1 looks good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---