[GitHub] [shardingsphere] sandynz opened a new issue, #19457: Extract job progress persistence from RuleAlteredJobSchedulerCenter and improve the implementation

[GitBox <gi...@apache.org>]

sandynz opened a new issue, #19457:
URL: https://github.com/apache/shardingsphere/issues/19457

   Part of #19421.
   
   Currently, all jobs progress persistence is in RuleAlteredJobSchedulerCenter, it's scheduled by ScheduledExecutorService with fixed delay. The persistence frequency is every 60 seconds before and 10 seconds now.
   ```
       private static final ScheduledExecutorService JOB_PERSIST_EXECUTOR = Executors.newSingleThreadScheduledExecutor(ExecutorThreadFactoryBuilder.build("scaling-job-persist-%d"));
       
       static {
           JOB_PERSIST_EXECUTOR.scheduleWithFixedDelay(new PersistJobContextRunnable(), 10, 10, TimeUnit.SECONDS);
       }
   ```
   
   It's not effient enough:
   - All jobs progresses will be persisted at the same time.
   - Scheduled by fixed delay, no matter there's new progress update or not, and it might cost much seconds to persist latest progress.
   
   Possible improvements:
   - Separate all jobs progresses persistence, every job use dedicated persistence, make sure progress could be persisted ASAP for job that has new progress, else no persistence for idle job;
   - Replace fixed delay, persist when there's job progress update ASAP; but we could not persist immediately once there's job progress update, since job progress might be updated very frequently, just persist to registry center might hurt performance; so it need a balance on persistence frequence, a suitable delay might be 2 seconds, it is configurable for every type of job if possible;
   
   Possible changes:
   - It might need observer pattern to capture new progress update, make sure underlying dumper and importer doesn't have job dedicated logic for common usage;
   - Every type of job could implement dedicated logic of job progress marsharling and unmarsharling;
   - Remove JOB_PERSIST_EXECUTOR and PersistJobContextRunnable in RuleAlteredJobSchedulerCenter;
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] sandynz commented on issue #19457: Extract job progress persistence from RuleAlteredJobSchedulerCenter and improve the implementation

[GitBox <gi...@apache.org>]

sandynz commented on issue #19457:
URL: https://github.com/apache/shardingsphere/issues/19457#issuecomment-1192122171

   @azexcy Could you help to follow it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] azexcy commented on issue #19457: Extract job progress persistence from RuleAlteredJobSchedulerCenter and improve the implementation

[GitBox <gi...@apache.org>]

azexcy commented on issue #19457:
URL: https://github.com/apache/shardingsphere/issues/19457#issuecomment-1192122749

   OK, I will follow it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] sandynz closed issue #19457: Extract job progress persistence from RuleAlteredJobSchedulerCenter and improve the implementation

[GitBox <gi...@apache.org>]

sandynz closed issue #19457: Extract job progress persistence from RuleAlteredJobSchedulerCenter and improve the implementation
URL: https://github.com/apache/shardingsphere/issues/19457


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org