You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "peilinqian (via GitHub)" <gi...@apache.org> on 2023/12/04 10:59:56 UTC

[I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]

peilinqian opened a new issue, #29276:
URL: https://github.com/apache/shardingsphere/issues/29276

   ## Bug Report
   
   **For English only**, other languages will not accept.
   
   Before report a bug, make sure you have:
   
   - Searched open and closed [GitHub issues](https://github.com/apache/shardingsphere/issues).
   - Read documentation: [ShardingSphere Doc](https://shardingsphere.apache.org/document/current/en/overview).
   
   Please pay attention on issues you submitted, because we maybe need more details. 
   If no response anymore and we cannot reproduce it on current information, we will **close it**.
   
   Please answer these questions before submitting your issue. Thanks!
   
   ### Which version of ShardingSphere did you use?
   ```
   /usr/bin/java
   we find java version: java8, full_version=1.8.0_242, full_path=/usr/bin/java
   ShardingSphere-5.4.2-SNAPSHOT
   Commit ID: dirty-dc9d8bef0e5696e38b9dec665fb4a39cd29c1062
   Commit Message: Improve stop_time display at show streaming list of CDC (#29090)
   Branch: master
   Build time: 2022-12-12T03:18:10+0800
   ```
   ### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
   ShardingSphere-Proxy
   shardingsphere-data-pipeline-cdc-client-5.4.2-SNAPSHOT.jar
   
   ### Expected behavior
   Two CDC jobs are executed concurrently, the first job is abnormal,the second job is normal.
   
   ### Actual behavior
   Two CDC jobs are executed concurrently, and all  jobs are abnormal
   ![image](https://github.com/apache/shardingsphere/assets/97432411/4397bcac-b19b-4320-9222-1de3aa9833ec)
   ![image](https://github.com/apache/shardingsphere/assets/97432411/4c086f8f-529f-4c8b-9259-0ef98a0ff7ae)
   
   **CDC server's log**
   ```
   [INFO ] 2023-12-04 18:56:55.730 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.c.prepare.CDCJobPreparer - initInventoryTasks cost 217 ms
   [INFO ] 2023-12-04 18:56:55.757 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
   [INFO ] 2023-12-04 18:56:55.759 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-1] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
   [INFO ] 2023-12-04 18:56:55.772 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
   [INFO ] 2023-12-04 18:56:55.777 [ShardingSphere-pipeline-Inventory-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-1] o.a.s.d.p.c.i.dumper.InventoryDumper - Inventory dump done, rowCount=0
   [WARN ] 2023-12-04 18:56:58.205 [epollEventLoopGroup-4-137] o.a.s.d.p.cdc.api.impl.CDCJobAPI - CDC job already exists in registry center, ignore, jobConfigKey=/pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.job.AbstractPipelineJob - stop tasks runner, jobId=j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@29d2dcdf, startTime=2023-12-04 18:56:55, cost 2463 ms
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter@7af53be7, startTime=2023-12-04 18:56:55, cost 2463 ms
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@848db32, startTime=2023-12-04 18:56:55, cost 2463 ms
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@5bcab474, startTime=2023-12-04 18:56:55, cost 2441 ms
   [INFO ] 2023-12-04 18:56:58.208 [epollEventLoopGroup-4-137] o.a.s.d.p.c.e.AbstractPipelineLifecycleRunnable - stop lifecycle executor org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper@6441bff6, startTime=2023-12-04 18:56:55, cost 2441 ms
   [WARN ] 2023-12-04 18:56:58.752 [epollEventLoopGroup-4-136] o.a.s.d.p.c.h.CDCBackendHandler - Could not find importer, ack id: ojL5PlfK_lRnSN9BDHNrilAvH
   [INFO ] 2023-12-04 18:57:00.229 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutting down.
   [INFO ] 2023-12-04 18:57:00.229 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED paused.
   [INFO ] 2023-12-04 18:57:00.245 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutdown complete.
   [ERROR] 2023-12-04 18:57:00.248 [ShardingSphere-pipeline-callback-0] o.a.s.d.p.c.task.TaskExecuteCallback - onFailure, task ID=ds_0.test_2_0#0
   java.lang.InterruptedException: sleep interrupted
           at java.lang.Thread.sleep(Native Method)
           at java.lang.Thread.sleep(Thread.java:340)
           at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
           at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
           at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   [ERROR] 2023-12-04 18:57:00.249 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - onFailure, inventory task execute failed.
   java.lang.InterruptedException: sleep interrupted
           at java.lang.Thread.sleep(Native Method)
           at java.lang.Thread.sleep(Thread.java:340)
           at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
           at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
           at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   [INFO ] 2023-12-04 18:57:00.277 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.job.AbstractPipelineJob - stop tasks runner, jobId=j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
   [INFO ] 2023-12-04 18:57:00.301 [Pipeline-EventListener-108] o.a.s.d.p.c.m.n.e.h.i.ConfigMetaDataChangedEventHandler - UPDATED job configuration: /pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config, disabled=false
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] o.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.3.2 created.
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] o.a.s.e.l.i.s.JobShutdownHookPlugin - Registering Quartz shutdown hook. j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.simpl.RAMJobStore - RAMJobStore initialized.
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.3.2) 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' with instanceId 'NON_CLUSTERED'
     Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
     NOT STARTED.
     Currently in standby mode.
     Number of jobs executed: 0
     Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
     Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
   
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' initialized from an externally provided properties instance.
   [INFO ] 2023-12-04 18:57:00.304 [epollEventLoopGroup-4-137] org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.3.2
   [INFO ] 2023-12-04 18:57:00.443 [ListenerNotify-j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9-0] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED started.
   [WARN ] 2023-12-04 18:57:00.489 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 already exists
   [INFO ] 2023-12-04 18:57:00.489 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - Execute job j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9
   [INFO ] 2023-12-04 18:57:00.492 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.pipeline.cdc.core.job.CDCJob - stopping true, ignore
   [INFO ] 2023-12-04 18:57:00.493 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - After j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 job execute
   [INFO ] 2023-12-04 18:57:00.540 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutting down.
   [INFO ] 2023-12-04 18:57:00.540 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED paused.
   [INFO ] 2023-12-04 18:57:00.550 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] org.quartz.core.QuartzScheduler - Scheduler j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_$_NON_CLUSTERED shutdown complete.
   [ERROR] 2023-12-04 18:57:00.575 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.e.e.h.g.LogJobErrorHandler - Job 'j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9' exception occur in job processing
   org.apache.shardingsphere.data.pipeline.core.exception.PipelineInternalException: java.util.concurrent.ExecutionException: java.lang.InterruptedException: sleep interrupted
           at org.apache.shardingsphere.data.pipeline.common.execute.ExecuteEngine.trigger(ExecuteEngine.java:138)
           at org.apache.shardingsphere.data.pipeline.cdc.core.job.CDCJob.executeInventoryTasks(CDCJob.java:149)
           at org.apache.shardingsphere.data.pipeline.cdc.core.job.CDCJob.execute(CDCJob.java:104)
           at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:33)
           at org.apache.shardingsphere.elasticjob.simple.executor.SimpleJobExecutor.process(SimpleJobExecutor.java:29)
           at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:173)
           at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.process(ElasticJobExecutor.java:142)
           at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:124)
           at org.apache.shardingsphere.elasticjob.executor.ElasticJobExecutor.execute(ElasticJobExecutor.java:100)
           at org.apache.shardingsphere.elasticjob.lite.internal.schedule.LiteJob.execute(LiteJob.java:35)
           at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
           at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
   Caused by: java.util.concurrent.ExecutionException: java.lang.InterruptedException: sleep interrupted
           at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
           at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
           at org.apache.shardingsphere.data.pipeline.common.execute.ExecuteEngine.trigger(ExecuteEngine.java:134)
           ... 11 common frames omitted
   Caused by: java.lang.InterruptedException: sleep interrupted
           at java.lang.Thread.sleep(Native Method)
           at java.lang.Thread.sleep(Thread.java:340)
           at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
           at org.apache.shardingsphere.data.pipeline.common.ingest.channel.memory.SimpleMemoryPipelineChannel.fetchRecords(SimpleMemoryPipelineChannel.java:63)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.doWithoutSorting(CDCImporter.java:96)
           at org.apache.shardingsphere.data.pipeline.cdc.core.importer.CDCImporter.runBlocking(CDCImporter.java:85)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.start(AbstractPipelineLifecycleRunnable.java:52)
           at org.apache.shardingsphere.data.pipeline.common.execute.AbstractPipelineLifecycleRunnable.run(AbstractPipelineLifecycleRunnable.java:83)
           at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   [INFO ] 2023-12-04 18:57:00.576 [Pipeline-EventListener-109] o.a.s.d.p.c.m.n.e.h.i.ConfigMetaDataChangedEventHandler - UPDATED job configuration: /pipeline/jobs/j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9/config, disabled=true
   [INFO ] 2023-12-04 18:57:00.577 [j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9_Worker-1] o.a.s.d.p.c.l.PipelineElasticJobListener - After j0302p0000d138c37ffa4ae8442cf4fd71bd3760c9 job execute
   ```
   
   
   ### Reason analyze (If you can)
   
   ### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
   
   ### Example codes for reproduce this issue (such as a github link).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]

Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on issue #29276:
URL: https://github.com/apache/shardingsphere/issues/29276#issuecomment-1860450979

   The elasticjob will take some time to shut down completely, so it's maybe not support concurrently now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29276:
URL: https://github.com/apache/shardingsphere/issues/29276#issuecomment-1896603412

   There hasn't been any activity on this issue recently, and in order to prioritize active issues, it will be marked as stale.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Two CDC jobs are executed concurrently, and all jobs are abnormal. [shardingsphere]

Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu closed issue #29276: Two CDC jobs are executed concurrently, and all jobs are abnormal.
URL: https://github.com/apache/shardingsphere/issues/29276


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org