You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ra...@apache.org on 2018/10/09 15:50:17 UTC
[15/45] carbondata git commit: [DOC] Add kafka example in StreamSQL
doc
[DOC] Add kafka example in StreamSQL doc
This closes #2769
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/629d6254
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/629d6254
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/629d6254
Branch: refs/heads/branch-1.5
Commit: 629d62549860e037c2816702564713ce696419d5
Parents: 54bcf49
Author: Jacky Li <ja...@qq.com>
Authored: Wed Sep 26 21:56:52 2018 +0800
Committer: QiangCai <qi...@qq.com>
Committed: Fri Sep 28 11:52:41 2018 +0800
----------------------------------------------------------------------
docs/streaming-guide.md | 33 +++++++++++++++++++++++++++------
1 file changed, 27 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata/blob/629d6254/docs/streaming-guide.md
----------------------------------------------------------------------
diff --git a/docs/streaming-guide.md b/docs/streaming-guide.md
index 3b71662..56e400e 100644
--- a/docs/streaming-guide.md
+++ b/docs/streaming-guide.md
@@ -304,8 +304,9 @@ Following example shows how to start a streaming ingest job
| register TIMESTAMP,
| updated TIMESTAMP
|)
- |STORED BY carbondata
+ |STORED AS carbondata
|TBLPROPERTIES (
+ | 'streaming'='source',
| 'format'='csv',
| 'path'='$csvDataDir'
|)
@@ -324,7 +325,7 @@ Following example shows how to start a streaming ingest job
| register TIMESTAMP,
| updated TIMESTAMP
|)
- |STORED BY carbondata
+ |STORED AS carbondata
|TBLPROPERTIES (
| 'streaming'='true'
|)
@@ -378,11 +379,14 @@ When this is issued, carbon will start a structured streaming job to do the stre
name STRING,
age INT
)
- STORED BY carbondata
+ STORED AS carbondata
TBLPROPERTIES(
- 'format'='socket',
- 'host'='localhost',
- 'port'='8888'
+ 'streaming'='source',
+ 'format'='socket',
+ 'host'='localhost',
+ 'port'='8888',
+ 'record_format'='csv', // can be csv or json, default is csv
+ 'delimiter'='|'
)
```
@@ -394,6 +398,7 @@ When this is issued, carbon will start a structured streaming job to do the stre
.format("socket")
.option("host", "localhost")
.option("port", "8888")
+ .option("delimiter", "|")
```
@@ -402,6 +407,22 @@ When this is issued, carbon will start a structured streaming job to do the stre
- In the given STMPROPERTIES, user must specify `'trigger'`, its value must be `ProcessingTime` (In future, other value will be supported). User should also specify interval value for the streaming job.
- If the schema specifid in sink table is different from CTAS, the streaming job will fail
+For Kafka data source, create the source table by:
+ ```SQL
+ CREATE TABLE source(
+ name STRING,
+ age INT
+ )
+ STORED AS carbondata
+ TBLPROPERTIES(
+ 'streaming'='source',
+ 'format'='kafka',
+ 'kafka.bootstrap.servers'='kafkaserver:9092',
+ 'subscribe'='test'
+ 'record_format'='csv', // can be csv or json, default is csv
+ 'delimiter'='|'
+ )
+ ```
##### STOP STREAM