You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/03/23 04:13:32 UTC

[GitHub] [incubator-doris] pigdance opened a new issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

pigdance opened a new issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557


   Reason: actual column number is less than schema column number. actual number: 1 sep: 	, schema number: 4; . src line: [{"data":[{"id":"26","name":"222","status":"22","date":"0000-00-00"}],"database":"test","es":1616467578000,"id":7,"isDdl":false,"mysqlType":{"id":"bigint(20)","name":"varchar(45)","status":"varchar(45)","date":"date"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":-5,"name":12,"status":12,"date":91},"table":"canal_test","ts":1616467578825,"type":"INSERT"}]; 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] hxianshun commented on issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

Posted by GitBox <gi...@apache.org>.
hxianshun commented on issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557#issuecomment-804677866


   > Reason: actual column number is less than schema column number. actual number: 1 sep: , schema number: 4; . src line: [{"data":[{"id":"26","name":"222","status":"22","date":"0000-00-00"}],"database":"test","es":1616467578000,"id":7,"isDdl":false,"mysqlType":{"id":"bigint(20)","name":"varchar(45)","status":"varchar(45)","date":"date"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":-5,"name":12,"status":12,"date":91},"table":"canal_test","ts":1616467578825,"type":"INSERT"}];
   
   u can use flink to preprocess your json data, and write into a new topic, then use Routine load to consume data.
   
   你可以用flink预处理下json数据,处理成doris认识的数据格式,然后写进一个新topic,再使用doris的Routine load导入 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] hf200012 commented on issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557#issuecomment-810104982


   > > Reason: actual column number is less than schema column number. actual number: 1 sep: , schema number: 4; . src line: [{"data":[{"id":"26","name":"222","status":"22","date":"0000-00-00"}],"database":"test","es":1616467578000,"id":7,"isDdl":false,"mysqlType":{"id":"bigint(20)","name":"varchar(45)","status":"varchar(45)","date":"date"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":-5,"name":12,"status":12,"date":91},"table":"canal_test","ts":1616467578825,"type":"INSERT"}];
   > 
   > u can use flink to preprocess your json data, and write into a new topic, then use Routine load to consume data.
   > 
   > 你可以用flink预处理下json数据,处理成doris认识的数据格式,然后写进一个新topic,再使用doris的Routine load导入
   
   After flink processing, directly use stream load to table


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] 812406210 commented on issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

Posted by GitBox <gi...@apache.org>.
812406210 commented on issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557#issuecomment-839526482


   > 写了个消费kafka的sink,然后批量insert 到doris里。
   你好,请问routine为啥消费kafka到doris没有数据呢,感觉是没有消费到一样


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] pigdance closed issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

Posted by GitBox <gi...@apache.org>.
pigdance closed issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] pigdance commented on issue #5557: 1、mysql开启binlog,写入kafka消息队列。但是doris 无法解析binlog的数据,有什么好的办法做mysql实时同步到doris吗?

Posted by GitBox <gi...@apache.org>.
pigdance commented on issue #5557:
URL: https://github.com/apache/incubator-doris/issues/5557#issuecomment-814687871


   写了个消费kafka的sink,然后批量insert 到doris里。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org