You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@seata.apache.org by "funky-eyes (via GitHub)" <gi...@apache.org> on 2024/11/15 07:37:14 UTC
[I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
funky-eyes opened a new issue, #7004:
URL: https://github.com/apache/incubator-seata/issues/7004
<!-- Please do not use this issue template to report security vulnerabilities but refer to our [security policy](https://github.com/seata/seata/security/policy). -->
- [ ] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.
### Ⅰ. Issue Description
908毫秒的时候全局事务已经在另一个线程里完成，910的时候还在回滚其中一个分支，明显的并行回滚导致，并且日志里回滚了2次6882078649837270974，该事务为一个时间较长的事务，时间大概为4分钟，导致定时任务会自动将rollbacking超过2分10秒的任务拉起来重试，而此时整好决议，所以会出现并发性回滚，而在raft下由于并发，会导致对应的globalsession已经被删除了，而接着发了一个branchsession操作相关的同步消息，导致出现npe
At 908 milliseconds, the global transaction was already completed in another thread, while at 910 milliseconds, one of the branches was still being rolled back. This clearly indicates parallel rollbacks. Additionally, the log shows that the transaction with ID 6882078649837270974 was rolled back twice. This transaction was a long-running one, lasting approximately 4 minutes, which caused the scheduled task to automatically retry tasks that had been in a 'rollbacking' state for more than 2 minutes and 10 seconds. By this time, the decision was already made, resulting in concurrent rollbacks. Under Raft, due to this concurrency, the corresponding global session was already deleted, and a branch session operation-related synchronization message was sent, which led to an NPE (NullPointerException)
```
2024-11-12 21:34:16.911 ERROR --- [JRaft-FSMCaller-Disruptor-0] [org.apache.seata.server.cluster.raft.RaftStateMachine]
[onExecuteRaft] []: Message synchronization failure: Cannot invoke "org.apache.seata.server.session.GlobalSession.getBranch(long)" because "globalSession" is null, msgType: RELEASE_BRANCH_SESSION_LOCK
==>
java.lang.NullPointerException: Cannot invoke "org.apache.seata.server.session.GlobalSession.getBranch(long)" because "globalSession" is null
at org.apache.seata.server.cluster.raft.execute.lock.BranchReleaseLockExecute.execute(BranchReleaseLockExecute.java:35)
at org.apache.seata.server.cluster.raft.execute.lock.BranchReleaseLockExecute.execute(BranchReleaseLockExecute.java:28)
at org.apache.seata.server.cluster.raft.RaftStateMachine.onExecuteRaft(RaftStateMachine.java:333)
at org.apache.seata.server.cluster.raft.RaftStateMachine.onApply(RaftStateMachine.java:174)
at com.alipay.sofa.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:597)
at com.alipay.sofa.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:561)
at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:467)
at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:73)
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:150)
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:142)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137)
at java.base/java.lang.Thread.run(Thread.java:1583)
<==
```
2024-11-12 21:34:16.892 INFO --- [SyncProcessing_1_1] [org.apache.seata.server.coordinator.DefaultCore] [lambda$doGlobalRollback$3] [193.193.193.37:8097:6882078649837270973]: Rollback branch transaction successfully, xid = 193.193.193.37:8097:6882078649837270973 branchId = 6882078649837270974
2024-11-12 21:34:16.908 INFO --- [SyncProcessing_1_1] [org.apache.seata.server.coordinator.DefaultCore] [doGlobalRollback] [193.193.193.37:8097:6882078649837270973]: Rollback global transaction successfully, xid = 193.193.193.37:8097:6882078649837270973.
2024-11-12 21:34:16.910 INFO --- [ServerHandlerThread_1_19_500] [org.apache.seata.server.coordinator.DefaultCore] [lambda$doGlobalRollback$3] [193.193.193.37:8097:6882078649837270973]: Rollback branch transaction successfully, xid = 193.193.193.37:8097:6882078649837270973 branchId = 6882078649837270974
### Ⅱ. Describe what happened
If there is an exception, please attach the exception trace:
```
Just paste your stack trace here!
```
### Ⅲ. Describe what you expected to happen
### Ⅳ. How to reproduce it (as minimally and precisely as possible)
1. xxx
2. xxx
3. xxx
Minimal yet complete reproducer code (or URL to code):
### Ⅴ. Anything else we need to know?
### Ⅵ. Environment:
- JDK version(e.g. `java -version`):
- Seata client/server version:
- Database version:
- OS(e.g. `uname -a`):
- Others:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "YongGoose (via GitHub)" <gi...@apache.org>.
YongGoose commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-2668778103
> > [@funky-eyes](https://github.com/funky-eyes)
> >
> > This issue sounds interesting! By the way, how is it prioritized? 🤔
>
> Are you referring to the priority between retry tasks and synchronous processing requests?
No, I was referring to the priority of the issue in the project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-2668754009
> [@funky-eyes](https://github.com/funky-eyes)
>
> This issue sounds interesting! By the way, how is it prioritized? 🤔
Are you referring to the priority between retry tasks and synchronous processing requests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "slievrly (via GitHub)" <gi...@apache.org>.
slievrly closed issue #7004: file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks
URL: https://github.com/apache/incubator-seata/issues/7004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-2670546396
> > > [@funky-eyes](https://github.com/funky-eyes)
> > > This issue sounds interesting! By the way, how is it prioritized? 🤔
> >
> >
> > Are you referring to the priority between retry tasks and synchronous processing requests?
>
> No, I was referring to the priority of the issue in the project.
The priority of this issue is not high for now, as the community has already come up with a locking mechanism to solve the transaction processing order issue. However, implementing this would significantly reduce throughput and performance. Therefore, the current approach is to handle exceptions caused by out-of-order processing at the code level. It can be considered that this issue is still in the discussion phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "zxxwyt (via GitHub) via notifications" <no...@seata.apache.org>.
zxxwyt commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-6624261960
I tested two complementary protections for the rollback overlap that remained after #7005. One coordinates the request handler and the recovery scan in the same TC process. The other prevents an in-flight rollback from writing Raft state after a leadership change.
### File mode: coordinate request and recovery rollback per XID
In the sequential rollback prototype, a process-local marker is shared by the normal rollback path and the scheduled recovery paths. The normal handler acquires the marker under the existing session lock before publishing `Rollbacking`. If the recovery scan finds the XID occupied, it skips it for that round. After acquiring the marker, the scanner reads the current session again and checks its status, since its earlier scan result may be stale. The executor releases the marker after rollback and synchronous cleanup.
I tested this with the actual File session store and a latch holding the first simulated RM response. While handler A waited, I ran recovery scan B on the same XID. The baseline sent **two rollback requests for the same branch**. With the marker, B returned and the request count stayed at **one**. After releasing A's response, A finished, the session was removed, and the marker could be acquired again.
### Raft mode: keep the term captured when rollback starts
Capture the leader term before the rollback begins and carry it through the execution, including the wait for the RM response. Set `Task.expectedTerm` on the subsequent Raft tasks for lock release, session changes and cleanup. Nested rollback calls keep the original term. JRaft checks it when admitting the task, so a TC that becomes leader again cannot submit an old response's cleanup under its new term.
I used three JVMs running the Seata coordinator and Raft session and lock stores, with real JRaft leadership transfers. I gated simulated RM replies and invoked the recovery entry point directly on B for the same XID and branch.
| Sequence | Observed result |
| --- | --- |
| A (term 1) waits for RM, B takes over in term 2, A's old reply arrives while A is a follower | The existing leader check rejected A's write. B completed normally once its reply was released. |
| A → B → A, without `expectedTerm` | B's reply was still held. A was leader again in term 3 when its term 1 reply arrived. The old execution released the lock and removed the session on all three peers. |
| A → B → A, with the captured `expectedTerm` | JRaft rejected the old release task with `expected_term=1 doesn't match current_term=3`. All peers retained the `Rollbacking` session, its branch and its lock. A fresh retry in term 3 completed cleanup on all peers. |
The Raft experiment used sequential AT rollback with Raft for both the session and lock stores. It exercised TC state and lock handling with a simulated RM response. The term protects writes across leader terms. Two executors in the same term can still overlap on the same XID. Per-XID coordination is still needed within one term. If it allows takeover within that term, a replicated per-XID execution generation checked when applying mutations is one option to reject stale writers. Already-dispatched RM requests still rely on mode-specific idempotency. Parallel branch handling also needs a defined point at which all in-flight work has finished before releasing an execution marker. Throughput impact remains to be measured.
The local prototype was based on `49e99de` with JRaft 1.3.14. The Raft and coordinator regression run had 167 passing tests and 5 skips.
中文简述：File 模式用同一 XID 的执行标记，让后台扫描跳过仍在回滚的事务。Raft 模式用回滚开始时的 leader term 保护后续写入。三节点实测 A→B→A 后，原代码允许 A 的旧回包释锁并清理会话；传入原 term 后，旧操作被拒绝，当前任期的新重试仍能完成。Raft term 保护跨任期写入，同任期内的竞争仍需按 XID 协调。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-2482031263
#7005 修复了并发出现的raft npe的问题，而二阶段的重试和决议可能会同时进行的问题还没处理
原方案 1. 增加本地锁，该方案在存算一体的raft和file下可解决，但是由于这种低概率事件而悲观的上锁，会导致不必要的性能损耗，并且在db和redis下依然无效
方案2. 增加动态的事务补偿时间，长事务的deadtime（认为事务在rollbaking和committing状态发生异常等需要异步任务进行补偿的时间间隔，默认2分10秒）可以通过globaltransactional注解进行指定每个事务粒度级别的deadtime，避免并发。（目前有全局可配置的server.retryDeadThreshold进行配置，但是粒度不够细），但是该方案缺点就是在db存储模式下需要增加表列，用户必须按照新的表接口变更后再升级server
方案3：共识算法，raft+db/redis等其它存储模式，来感知决议节点是否已下线，补偿任务仅补偿对应xid的server已经下线的rollbacking状态的事务，因为对应xid的server在线，不应该再进行补偿该事务，因为如果同步的过程中出现异常，事务会changestatus，并不会保持在rollbacking，也就是如果xid对应的server存活，rollbacking只可能在存活节点上正在运行，而不需要补偿。
#7005 fixed the Raft NPE issue caused by concurrency, but the issue where two-phase retries and decisions might occur simultaneously has not been addressed yet.
Original Plan:
Add local locks: This solution resolves the issue in Raft with integrated storage and computation (such as Raft and file systems), but introducing pessimistic locking due to such low-probability events can cause unnecessary performance overhead. Additionally, this solution remains ineffective in DB and Redis environments.
Add dynamic transaction compensation time: The deadtime for long transactions (defined as the interval during which a transaction might encounter anomalies in the rollback or committing state and needs an asynchronous compensation task) is set to 2 minutes and 10 seconds by default. The globaltransactional annotation can be used to specify the deadtime for each transaction at a granular level to avoid concurrency. (Currently, there is a globally configurable server.retryDeadThreshold, but its granularity is insufficient.) However, the drawback of this solution is that, in DB storage mode, it requires adding new table columns, and users must update the server after modifying the table interface.
Consensus algorithm (Raft + DB/Redis and other storage modes): This solution involves using Raft and storage modes like DB/Redis to detect whether the decision-making node is offline. The compensation task should only compensate for transactions in the "rollbacking" state on servers corresponding to an offline xid. If the server corresponding to the xid is online, no compensation is needed for that transaction, because if an exception occurs during synchronization, the transaction will change its status and will not remain in the "rollbacking" state. Therefore, if the server corresponding to the xid is alive, the "rollbacking" state can only be running on the live node, and no compensation is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] file&raft doGlobalRollback and doGlobalCommit may have concurrency issues with retry tasks [incubator-seata]
Posted by "YongGoose (via GitHub)" <gi...@apache.org>.
YongGoose commented on issue #7004:
URL: https://github.com/apache/incubator-seata/issues/7004#issuecomment-2658301893
@funky-eyes
This issue sounds interesting!
By the way, how is it prioritized? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org