You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2018/12/31 05:52:09 UTC
[GitHub] tuteng opened a new pull request #3268: Optimization of binlog to
pulsar use canal
tuteng opened a new pull request #3268: Optimization of binlog to pulsar use canal
URL: https://github.com/apache/pulsar/pull/3268
### Motivation
Optimization of mysql binlog to pulsar use canal
### Modifications
Add MessageUtils for parse columns
Support query of presto sql
### Result
#### INSERT
mysql:
```
MariaDB [aaa]> insert into users321(name, extra) values('xxxddxxxmm', 'ddd');
```
python consumer:
```
{u'timestamp': u'2018-12-31 00:31:24', u'message': u'[{"data":null,"database":"","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"insert into users321(name, extra) values(\'xxxddxxxmm\', \'ddd\')","sqlType":null,"table":"","ts":1546234284669,"type":"QUERY"},{"data":[{"isKey":"1","isNull":"0","index":"0","mysqlType":"int(11)","columnName":"id","columnValue":"34","updated":"1"},{"isKey":"0","isNull":"0","index":"1","mysqlType":"varchar(50)","columnName":"name","columnValue":"xxxddxxxmm","updated":"1"},{"isKey":"0","isNull":"0","index":"2","mysqlType":"varchar(50)","columnName":"extra","columnValue":"ddd","updated":"1"}],"database":"aaa","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"","sqlType":null,"table":"users321","ts":1546234284670,"type":"INSERT"}]', u'id': 58}
```
presto sql:
```
presto> select id, timestamp, message from pulsar."public/default".my_topic_test where id=58;
id | timestamp |
----+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
58 | 2018-12-31 00:31:24 | [{"data":null,"database":"","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"insert into users321(name, extra) values('xxxddxxxmm', 'ddd')","sqlType"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services