You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by vesense <gi...@git.apache.org> on 2016/11/01 03:33:09 UTC
[GitHub] storm pull request #1754: [STORM-2177][SQL] Support TSV as input / output fo...
GitHub user vesense opened a pull request:
https://github.com/apache/storm/pull/1754
[STORM-2177][SQL] Support TSV as input / output format
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vesense/storm STORM-2177
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1754.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1754
----
commit 67178e00b1ef9b07367728bef2632009762d9b0c
Author: Xin Wang <be...@163.com>
Date: 2016-10-29T15:59:56Z
[STORM-2172][SQL] Support Avro as input / output format
commit 2d1785b3bb47c103906a24a40cc1a3b4ffcd70fc
Author: Xin Wang <be...@163.com>
Date: 2016-10-31T15:27:05Z
[STORM-2177][SQL] Support TSV as input / output format
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as input / ...
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1754
@HeartSaVioR Do you have time to take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as input / ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1754
@vesense
Changeset looks good. I'll do some manual tests later today or tomorrow. Thanks for nice work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as input / ...
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1754
The code is ready for reviewing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as input / ...
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1754
I added STORM-2173 to current PR since it is on top of #1751 too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][SQL] Support TSV as input / output format
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1754
This PR is on top of https://github.com/apache/storm/pull/1751
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as input / ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1754
CSV -> TSV
```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY INT) STORED AS INPUTFORMAT 'org.apache.storm.sql.runtime.serde.csv.CsvScheme' OUTPUTFORMAT 'org.apache.storm.sql.runtime.serde.csv.CsvSerializer' LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES '{ "producer": { "bootstrap.servers": "localhost:9092", "acks": "1", "key.serializer": "org.apache.storm.kafka.IntSerializer", "value.serializer": "org.apache.storm.kafka.ByteBufferSerializer" } }'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) STORED AS INPUTFORMAT 'org.apache.storm.sql.runtime.serde.tsv.TsvScheme' OUTPUTFORMAT 'org.apache.storm.sql.runtime.serde.tsv.TsvSerializer' LOCATION 'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES ' { "producer": { "bootstrap.servers": "localhost:9092", "acks": "1", "key.serializer": "org.apache.storm.kafka.IntSerializer", "value.serializer": "org.apache.storm.kafka.ByteBufferSerializer" }, "input.tsv.delimiter": "\t", "output.tsv.delimiter": "\t"}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```
TSV -> CSV
```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY INT) STORED AS INPUTFORMAT 'org.apache.storm.sql.runtime.serde.tsv.TsvScheme' OUTPUTFORMAT 'org.apache.storm.sql.runtime.serde.tsv.TsvSerializer' LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES '{ "producer": { "bootstrap.servers": "localhost:9092", "acks": "1", "key.serializer": "org.apache.storm.kafka.IntSerializer", "value.serializer": "org.apache.storm.kafka.ByteBufferSerializer" }, "input.tsv.delimiter": "|", "output.tsv.delimiter": "|" }'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) STORED AS INPUTFORMAT 'org.apache.storm.sql.runtime.serde.csv.CsvScheme' OUTPUTFORMAT 'org.apache.storm.sql.runtime.serde.csv.CsvSerializer' LOCATION 'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES ' { "producer": { "bootstrap.servers": "localhost:9092", "acks": "1", "key.serializer": "org.apache.storm.kafka.IntSerializer", "value.serializer": "org.apache.storm.kafka.ByteBufferSerializer" }}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```
Confirmed both are working well. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1754: [STORM-2177][STORM-2173][SQL] Support TSV/CSV as i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1754
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---