You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "wjdxw (via GitHub)" <gi...@apache.org> on 2024/08/13 03:34:15 UTC
[I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
wjdxw opened a new issue, #2429:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429
### Which version of ElasticJob did you use?
3.0.3
服务器A 和服务器B (服务器A和B 至少保证一个是在线状态)
服务器A运行job分片0 服务器B运行job分片1
操作:step:1 job 分片0和1 正在running,停止并启动服务器B,此时job 分片1的任务会转移在服务器A上执行。
step2:等待服务器B启动成功,此时服务器A上正在运行失效转移的job分片1,此时服务器A再次启停
### Expected behavior
job 到点能够正常触发
### Actual behavior
job 不在触发,几乎是必现
### Reason analyze (If you can)
zk节点查看状态
分片1 的节点 failover 和failovering 及running 状态一直存在,一直未清除导致job 不再执行。
应该是停止服务器A的时候,那块有点问题
当step2 中服务器A启动的时候,LegacyCrashedRunningItemListener 无法做监听操作,因为一直会有两个实例在,代码bu hui qu不会去做失效转移相关操作
<img width="1137" alt="image" src="https://github.com/user-attachments/assets/e0e78395-9036-4850-8d78-f0a138628c5d">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
Posted by "wjdxw (via GitHub)" <gi...@apache.org>.
wjdxw commented on issue #2429:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429#issuecomment-2288828405
看下这三点优化,如果有必要,我这边提交PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
Posted by "linghengqian (via GitHub)" <gi...@apache.org>.
linghengqian commented on issue #2429:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429#issuecomment-2288900718
I will say that I am not actually the original author of this piece of logic, and from my perspective there is nothing wrong with the revised opinion. PR is always welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
Posted by "linghengqian (via GitHub)" <gi...@apache.org>.
linghengqian closed issue #2429: 失效转移导致job不在执行
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
Posted by "linghengqian (via GitHub)" <gi...@apache.org>.
linghengqian commented on issue #2429:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429#issuecomment-2286458194
- Do you want to submit a fix PR that includes unit tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 失效转移导致job不在执行 [shardingsphere-elasticjob]
Posted by "wjdxw (via GitHub)" <gi...@apache.org>.
wjdxw commented on issue #2429:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2429#issuecomment-2288825840
找到真实原因了
**A节点宕机 B 节点监听 相关失效转移节点操作**
1.找到服务器A sharding/{分片}/running
public void setCrashedFailoverFlag(final int item) {
if (!isFailoverAssigned(item)) {
jobNodeStorage.createJobNodeIfNeeded(FailoverNode.getItemsNode(item));
jobNodeStorage.removeJobNodeIfExisted(ShardingNode.getRunningNode(item));
}
}
sharing/{分片}/failover 节点不存在,创建/leader/failover/items//{分片}/ 删除 sharding//{分片}/running
failoverIfNecessary()
/leader/failover/items 节点存在 && /leader/failover/items 当前节点需要有子节点
当前实例的job 不在运行状态
/leader/failover/latch 节点选主成功后,执行FailoverLeaderExecutionCallback回调操作
needFailover()
创建 sharding/{分片}/failover
创建 sharding/{分片}/failovering
删除 leader/failover/items/{分片}
执行execute()
此时B节点正在执行当前分片的实效转移,执行过程中
创建sharding/{分片}/running
服务器A 重启成功
由于job执行时间较长
此时服务器B重启,服务器B宕机,使用kill-9 后面的都不会执行
###
(_等待执行完成之后
删除 sharding/{分片}/failover
删除 sharding/{分片}/failovering
删除 创建sharding/{分片}/running_ )不执行
**B节点宕机,A节点监听**
1.拿到sharding/{分片}/failovering
相关的信息进行失效转移
public void setCrashedFailoverFlagDirectly(final int item) {
jobNodeStorage.createJobNodeIfNeeded(FailoverNode.getItemsNode(item));
}
这边的逻辑不会去删除running 节点,对于异常宕机的服务器,可能就会导致一些节点无法删除
执行execute()
此时正常A节点和B节点 都应该继续执行job,但是却没有执行
因为sharding/{分片}/running 这个节点一直存在,在执行代码的时候会执行这一段代码 waitingOtherShardingItemCompleted();
private void waitingOtherShardingItemCompleted() {
while (executionService.hasRunningItems()) {
log.debug("Job '{}' sleep short time until other job completed.", jobName);
BlockUtils.waitingShortTime();
}
}
这边会判断是否有running 的节点,实际上这边有running的节点一直没有消失,所以代码一直会阻塞在这里,导致无法进行执行,如果是此异常,可以通过删除sharding/{分片}/running,这样就可以继续执行了,执行之后也会正常
删除 sharding/{分片}/failover
删除 sharding/{分片}/failovering
Job正常还有启动的时候的监听,继续如上B节点宕机,B节点也会启动,B节点启动之后做了什么操作
**B节点启动,A节点监听**
如果只有一个实例时
获取 sharding/{分片}/failovering ,如果执行当前failering的实例不可以用,删除sharding/{分片}/failovering
获取 sharing/分片/running,如果执行当前的running的实例不可用,删除删除sharding/{分片}/failovering
源码在的判断是否只有一个实例的时候,由于生产的实例A节点在,B节点启动的时候就试两个,代码永远走不到下面,所以这个runinng的节点一直存在,就导致一直无法执行
所以源码这个判断逻辑有三个优化点:
1. 在执行waitingOtherShardingItemCompleted 方法的时候,hasRunningItems 这边也应该判断下 running 的节点是否是在可用的节点上,不然无效的节点running 也会造成阻塞
2.setCrashedFailoverFlagDirectly 的时候,如果一些异常宕机,造成runnning 和failovering节点在的时候,这边就不会清理runinng 节点
3.isTheOnlyInstance 这个方法是否可以去掉,对于生产,不会只有一个实例存在的情况,除非第一次第一个实例上生产。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org