You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "peilinqian (via GitHub)" <gi...@apache.org> on 2023/12/04 10:59:56 UTC
[I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]
peilinqian opened a new issue, #29276:
URL: https://github.com/apache/shardingsphere/issues/29276
## Bug Report
**For English only**, other languages will not accept.
Before report a bug, make sure you have:
- Searched open and closed [GitHub issues](https://github.com/apache/shardingsphere/issues).
- Read documentation: [ShardingSphere Doc](https://shardingsphere.apache.org/document/current/en/overview).
Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot reproduce it on current information, we will **close it**.
Please answer these questions before submitting your issue. Thanks!
### Which version of ShardingSphere did you use?
```
/usr/bin/java
we find java version: java8, full_version=1.8.0_242, full_path=/usr/bin/java
ShardingSphere-5.4.2-SNAPSHOT
Commit ID: dirty-dc9d8bef0e5696e38b9dec665fb4a39cd29c1062
Commit Message: Improve stop_time display at show streaming list of CDC (#29090)
Branch: master
Build time: 2022-12-12T03:18:10+0800
```
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy
shardingsphere-data-pipeline-cdc-client-5.4.2-SNAPSHOT.jar
### Expected behavior
Two CDC jobs are executed concurrently, the first job is abnormal,the second job is normal.
### Actual behavior
Two CDC jobs are executed concurrently, and all jobs are abnormal


**CDC server's log**
```
[INFO ] 2023-12-04 18:56:55.730 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.c.prepare.CDCJobPreparer - initInventoryTasks cost 217 ms
[INFO ] 2023-12-04 18:56:55.757 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
[INFO ] 2023-12-04 18:56:55.759 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-1] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
[INFO ] 2023-12-04 18:56:55.772 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
[INFO ] 2023-12-04 18:56:55.777 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-1] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
[WARN ] 2023-12-04 18:56:58.205 [epollEventLoopGroup-4-137] o.a.s.d.p.cdc.api.impl.CDCJobAPI - CDC job already exists in registry center, ignore, jobConfigKey=/pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.job.AbstractPipelineJob - stop tasks runner, jobId=j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@29d2dcdf, startTime=2023-12-04 18:56:55, cost 2463 ms
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter@7af53be7, startTime=2023-12-04 18:56:55, cost 2463 ms
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@848db32, startTime=2023-12-04 18:56:55, cost 2463 ms
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@5bcab474, startTime=2023-12-04 18:56:55, cost 2441 ms
[INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@6441bff6, startTime=2023-12-04 18:56:55, cost 2441 ms
[WARN ] 2023-12-04 18:56:58.752 [epollEventLoopGroup-4-136] o.a.s.d.p.c.h.CDCBackendHandler - Could not find importer, ack id: ojL5PlfK_lRnSN9BDHNrilAvH
[INFO ] 2023-12-04 18:57:00.229 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutting down.
[INFO ] 2023-12-04 18:57:00.229 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED paused.
[INFO ] 2023-12-04 18:57:00.245 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutdown complete.
[ERROR] 2023-12-04 18:57:00.248 [ShardingSphere-pipeline-callback-0] o.a.s.d.p.c.task.TaskExecuteCallback - onFailure, task ID=ds_0.test_2_0#0
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[ERROR] 2023-12-04 18:57:00.249 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - onFailure, inventory task execute failed.
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] 2023-12-04 18:57:00.277 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.job.AbstractPipelineJob - stop tasks runner, jobId=j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
[INFO ] 2023-12-04 18:57:00.301 [Pipeline-EventListener-108] o.a.s.d.p.c.m.n.e.h.i.ConfigMetaDataChangedEventHandler - UPDATED job configuration: /pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config, disabled=false
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] o.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.3.2 created.
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] o.a.s.e.l.i.s.JobShutdownHookPlugin - Registering Quartz shutdown hook. j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.simpl.RAMJobStore - RAMJobStore initialized.
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.3.2) 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' initialized from an externally provided properties instance.
[INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.3.2
[INFO ] 2023-12-04 18:57:00.443 [ListenerNotify-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED started.
[WARN ] 2023-12-04 18:57:00.489 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 already exists
[INFO ] 2023-12-04 18:57:00.489 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - Execute job j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
[INFO ] 2023-12-04 18:57:00.492 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - stopping true, ignore
[INFO ] 2023-12-04 18:57:00.493 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - After j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 job execute
[INFO ] 2023-12-04 18:57:00.540 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutting down.
[INFO ] 2023-12-04 18:57:00.540 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED paused.
[INFO ] 2023-12-04 18:57:00.550 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutdown complete.
[ERROR] 2023-12-04 18:57:00.575 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.e.e.h.g.LogJobErrorHandler - Job 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' exception occur in job processing
org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: java.util.concurrent.ExecutionException: java.lang.InterruptedException: sleep interrupted
at org.apache.shardingsphere.data.pipeline.common.execute.ExecuteEngine.trigger(ExecuteEngine.java:138)
at org.apache.shardingsphere.data.pipeline.cdc.core.job.CDCJob.executeInventoryTasks(CDCJob.java:149)
at org.apache.shardingsphere.data.pipeline.cdc.core.job.CDCJob.execute(CDCJob.java:104)
at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:33)
at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:29)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:173)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:142)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:124)
at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:100)
at org.apache.shardingsphere.elasticjob.lite.internal.schedule.LiteJob.execute(LiteJob.java:35)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.util.concurrent.ExecutionException: java.lang.InterruptedException: sleep interrupted
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.shardingsphere.data.pipeline.common.execute.ExecuteEngine.trigger(ExecuteEngine.java:134)
... 11 common frames omitted
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] 2023-12-04 18:57:00.576 [Pipeline-EventListener-109] o.a.s.d.p.c.m.n.e.h.i.ConfigMetaDataChangedEventHandler - UPDATED job configuration: /pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config, disabled=true
[INFO ] 2023-12-04 18:57:00.577 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - After j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 job execute
```
### Reason analyze (If you can)
### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
### Example codes for reproduce this issue (such as a github link).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on issue #29276:
URL: https://github.com/apache/shardingsphere/issues/29276#issuecomment-1860450979
The elasticjob will take some time to shut down completely, so it's maybe not support concurrently now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29276:
URL: https://github.com/apache/shardingsphere/issues/29276#issuecomment-1896603412
There hasn't been any activity on this issue recently, and in order to prioritize active issues, it will be marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu closed issue #29276: Two CDC jobs are executed concurrently, and all jobs are abnormal.
URL: https://github.com/apache/shardingsphere/issues/29276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org