You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "GuiwenChen (via GitHub)" <gi...@apache.org> on 2025/04/28 14:10:07 UTC
[I] During data migration, an error was reported in the incremental migration process. [shardingsphere]
GuiwenChen opened a new issue, #35285:
URL: https://github.com/apache/shardingsphere/issues/35285
## Bug Report
### Which version of ShardingSphere did you use?
5.5.1
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy
### Expected behavior
Normally subscribe to and parse MySQL's binlog.
### Actual behavior
```
[ERROR] 2025-04-28 21:11:10.219 [j0102p0000909ead799d8d661937e23e5b3160fd8f_Worker-1] o.a.s.e.e.h.g.LogJobErrorHandler - Job 'j0102p0000909ead799d8d661937e23e5b3160fd8f' exception occur in job processing
org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: java.util.concurrent.ExecutionException: org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: Unknown system variable '@MASTER_BINLOG_CHECKSUM'
at org.apache.shardingsphere.data.pipeline.core.execute.PipelineExecuteEngine.trigger(PipelineExecuteEngine.java:138)
at org.apache.shardingsphere.data.pipeline.core.task.runner.TransmissionTasksRunner.executeIncrementalTasks(TransmissionTasksRunner.java:112)
at org.apache.shardingsphere.data.pipeline.core.task.runner.TransmissionTasksRunner.access$500(TransmissionTasksRunner.java:49)
at org.apache.shardingsphere.data.pipeline.core.task.runner.TransmissionTasksRunner$InventoryTaskExecuteCallback.onSuccess(TransmissionTasksRunner.java:141)
at org.apache.shardingsphere.data.pipeline.core.execute.PipelineExecuteEngine.trigger(PipelineExecuteEngine.java:141)
at org.apache.shardingsphere.data.pipeline.core.task.runner.TransmissionTasksRunner.executeInventoryTasks(TransmissionTasksRunner.java:91)
at org.apache.shardingsphere.data.pipeline.core.task.runner.TransmissionTasksRunner.start(TransmissionTasksRunner.java:78)
at org.apache.shardingsphere.data.pipeline.core.job.executor.DistributedPipelineJobExecutor.execute(DistributedPipelineJobExecutor.java:110)
at org.apache.shardingsphere.data.pipeline.core.job.executor.DistributedPipelineJobExecutor.execute(DistributedPipelineJobExecutor.java:80)
at org.apache.shardingsphere.data.pipeline.scenario.migration.MigrationJob.execute(MigrationJob.java:43)
at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:33)
at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:29)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:173)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:142)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:124)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:100)
at org.apache.shardingsphere.elasticjob.lite.internal.schedule.LiteJob.execute(LiteJob.java:35)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.util.concurrent.ExecutionException: org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: Unknown system variable '@MASTER_BINLOG_CHECKSUM'
at java.base/java.util.concurrent.CompletableFuture.wrapInExecutionException(CompletableFuture.java:345)
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:440)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2117)
at org.apache.shardingsphere.data.pipeline.core.execute.PipelineExecuteEngine.trigger(PipelineExecuteEngine.java:134)
... 18 common frames omitted
Caused by: org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: Unknown system variable '@MASTER_BINLOG_CHECKSUM'
at org.apache.shardingsphere.data.pipeline.mysql.ingest.incremental.client.MySQLBinlogClient.waitExpectedResponse(MySQLBinlogClient.java:260)
at org.apache.shardingsphere.data.pipeline.mysql.ingest.incremental.client.MySQLBinlogClient.execute(MySQLBinlogClient.java:130)
at org.apache.shardingsphere.data.pipeline.mysql.ingest.incremental.client.MySQLBinlogClient.initDumpConnectSession(MySQLBinlogClient.java:186)
at org.apache.shardingsphere.data.pipeline.mysql.ingest.incremental.client.MySQLBinlogClient.subscribe(MySQLBinlogClient.java:178)
at org.apache.shardingsphere.data.pipeline.mysql.ingest.incremental.dumper.MySQLIncrementalDumper.runBlocking(MySQLIncrementalDumper.java:100)
at org.apache.shardingsphere.data.pipeline.core.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:50)
at org.apache.shardingsphere.data.pipeline.core.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:81)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1848)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)
```
### Reason analyze (If you can)
### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
If the migration starts from the original MySQL, the migration process does not report any errors. However, if the migration is performed from another ShardingSphere-Proxy, the full migration succeeds, but the incremental migration encounters the aforementioned error. What could be causing this?
### Example codes for reproduce this issue (such as a github link).
source Shardingsphere-Proxy database-sharding.yaml
```
databaseName: account_system1
dataSources:
ds:
url: jdbc:mysql://127.0.0.1:3306/account_system1?useSSL=false&allowPublicKeyRetrieval=true&characterEncoding=UTF-8
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
rules:
- !SINGLE
tables:
- "*.*"
- !SHARDING
tables:
sync:
actualDataNodes: ds.sync,ds.sync_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: sync_inline
orders:
actualDataNodes: ds.orders,ds.orders_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: orders_inline
order_item:
actualDataNodes: ds.order_item,ds.order_item_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: order_item_inline
order_status:
actualDataNodes: ds.order_status,ds.order_status_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: order_status_inline
log_main:
actualDataNodes: ds.log_main,ds.log_main_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: log_main_inline
log_details:
actualDataNodes: ds.log_details,ds.log_details_${0..2}
tableStrategy:
standard:
shardingColumn: company_id
shardingAlgorithmName: log_details_inline
bindingTables:
- sync,orders,order_item,order_status,log_main,log_details
defaultTableStrategy:
none:
shardingAlgorithms:
sync_inline:
type: INLINE
props:
algorithm-expression: sync_${company_id.intdiv(50)}
orders_inline:
type: INLINE
props:
algorithm-expression: orders_${company_id.intdiv(50)}
order_item_inline:
type: INLINE
props:
algorithm-expression: order_item_${company_id.intdiv(50)}
order_status_inline:
type: INLINE
props:
algorithm-expression: order_status_${company_id.intdiv(50)}
log_main_inline:
type: INLINE
props:
algorithm-expression: log_main_${company_id.intdiv(50)}
log_details_inline:
type: INLINE
props:
algorithm-expression: log_details_${company_id.intdiv(50)}
keyGenerators:
snowflake:
type: SNOWFLAKE
```
target Shardingsphere-Proxy database-sharding.yaml
```
databaseName: account_system_merge
dataSources:
ds:
url: jdbc:mysql://127.0.0.1:3306/account_system_merge?useSSL=false&allowPublicKeyRetrieval=true&characterEncoding=UTF-8
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
rules:
- !SINGLE
tables:
- "*.*"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] During data migration, an error was reported in the incremental migration process. [shardingsphere]
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang commented on issue #35285:
URL: https://github.com/apache/shardingsphere/issues/35285#issuecomment-2876314773
Hi @GuiwenChen , what is the version of MySQL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] During data migration, an error was reported in the incremental migration process. [shardingsphere]
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang commented on issue #35285:
URL: https://github.com/apache/shardingsphere/issues/35285#issuecomment-2876320639
Hi @GuiwenChen , Proxy cannot provide binlog, so it cannot support the scenario you describe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] During data migration, an error was reported in the incremental migration process. [shardingsphere]
Posted by "GuiwenChen (via GitHub)" <gi...@apache.org>.
GuiwenChen commented on issue #35285:
URL: https://github.com/apache/shardingsphere/issues/35285#issuecomment-2876528734
Okay, if ShardingSphere is already in use, what are the options to support re-sharding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] During data migration, an error was reported in the incremental migration process. [shardingsphere]
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang closed issue #35285: During data migration, an error was reported in the incremental migration process.
URL: https://github.com/apache/shardingsphere/issues/35285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org