[GitHub] [incubator-doris] xy720 commented on issue #6287: [Proposal] Support synchronizing MySQL binlog in real time

[GitBox <gi...@apache.org>]

xy720 commented on issue #6287:
URL: https://github.com/apache/incubator-doris/issues/6287#issuecomment-884136384


   We already finish stage 1 in pr 
   https://github.com/apache/incubator-doris/pull/6289
   
   The proformance we test is below:
   
   ## Environment
   
   |  Cores   | Memory  |
   |  ----  | ----  |
   | 96 | 360G |
   
   ## Test Scenarios
   
   1、Use mysqlslap as tool of MySQL for pressure testing.
   
   ```
   Three source tables with the same structure in MySQL database (t1, t2, t3): 
   (avg row size 35 bytes)
   +-------+---------+------+-----+---------+-------+
   | Field | Type    | Null | Key | Default | Extra |
   +-------+---------+------+-----+---------+-------+
   | a     | int(11) | NO   | PRI | NULL    |       |
   | b     | int(11) | YES  |     | NULL    |       |
   +-------+---------+------+-----+---------+-------+
   ```
   
   ```
   Three target tables of Doris database (test1, test2, test3)
   +-------+------+------+-------+---------+---------+
   | Field | Type | Null | Key   | Default | Extra   |
   +-------+------+------+-------+---------+---------+
   | a     | INT  | No   | true  | NULL    |         |
   | b     | INT  | No   | false | NULL    | REPLACE |
   +-------+------+------+-------+---------+---------+
   ```
   
   ## Test Resule
   
   ### scene 1
   
   1. start-up canal
   2. create sync job and start synchronizing
   3. Perform insert operations in three source tables in Mysql
   
   |  stage  |  statement  |  data size  |  cost(s)  |  tps avg  |  speed  |
   |  ----  |  ----  | ---- |  ----  |  ----  |  ----  |
   | mysql insert | 1000w   |  120M  |  74  |  135135  |  1.6M/s |
   | doris synchronization   | 1000w  |  120M  |  81  |  123457  |  1.5M/s  |
   
   #### scene 2
   
   1. start-up canal
   2. Perform insert operations in three source tables in Mysql
   3. create sync job and start synchronizing
   
   |  stage  |  statement  |  data size  |  cost(s)  |  tps avg  |  speed  |
   |  ----  |  ----  | ---- | ----  | ----  |  ----  |
   | mysql insert | 1000w   |  120M  |  76  | 131578  |  0.6M/s |
   | doris synchronization  | 1000w  |  120M  |  72  | 138889  |  1.7M/s  |
   
   
   ### Scene 3
   
   1. create sync job and start synchronizing
   2. Delete data from three source tables
   
   |  stage  |  op  |  row  |  data size  |  cost(s)  |  tps avg  |  speed  |
   |  ----  |  ----  |  ----  |  ----  | ---- | ----  |  ----  |
   | mysql delete | delete   | 10000000   |  120M  |  30  |   33w  |  4M/s |
   | doris synchronization | delete   | 10000000  |  120M  |  35  |  30w  |  3.4M/s  |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org