You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Vladislav Pyatkov (Jira)" <ji...@apache.org> on 2022/12/30 13:58:00 UTC
[jira] [Updated] (IGNITE-15568) Striped Disruptor doesn't work with JRaft event handlers properly
[ https://issues.apache.org/jira/browse/IGNITE-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladislav Pyatkov updated IGNITE-15568:
---------------------------------------
Description:
The following scenario is broken:
# Two raft groups are started and mapped to the same stripe.
# Two LogEntryAndClosure events are added in quick succession so they form distruptor batch: first for group 1, second for group 2.
First event is delivered to group 1 with endOfBatch=false, so it's cached in org.apache.ignite.raft.jraft.core.NodeImpl.LogEntryAndClosureHandler#tasks and is not processed.
Second event is delivered to group 2 with endOfBatch=true and processed, but first event will remain in queue unprocessed forever, because LogEntryAndClosureHandler are different instances per raft group.
The possible WA for this is to set org.apache.ignite.raft.jraft.option.RaftOptions#applyBatch=1
Reproducible by org.apache.ignite.internal.table.TxDistributedTest_1_1_1#testCrossTable + applyBatch=32 in ignite-15085 branch
*Implementation notes*
My proposal goes bound Disruptor. The striped disruptor implementation has an interceptor that proposes an event to a specific interceptor. Only the last event in the batch has a completion batch flag. For the other RAFT groups, which has been notified in the striped disruptor, required to create an event to fix a batch into the specific group. The new event will be created in the common striped disruptor interceptor, and it will send to a specific interceptor with flag about batch completion.
The rule of handling the new event is differenced for various interceptor:
{code:java|title=title=ApplyTaskHandler (FSMCallerImpl#runApplyTask)}
if (maxCommittedIndex >= 0) {
doCommitted(maxCommittedIndex);
return -1;
}
{code}
{code:java|title=LogEntryAndClosureHandler(LogEntryAndClosureHandler#onEvent)}
if (this.tasks.size() > 0) {
executeApplyingTasks(this.tasks);
this.tasks.clear();
}
{code}
{code:java|title=ReadIndexEventHandler(ReadIndexEventHandler#onEvent)}
if (this.events.size() > 0) {
executeReadIndexEvents(this.events);
this.events.clear();
}
{code}
{code:java|title=StableClosureEventHandler(StableClosureEventHandler#onEvent)}
if (this.ab.size > 0) {
this.lastId = this.ab.flush();
setDiskId(this.lastId);
}
{code}
was:
The following scenario is broken:
# Two raft groups are started and mapped to the same stripe.
# Two LogEntryAndClosure events are added in quick succession so they form distruptor batch: first for group 1, second for group 2.
First event is delivered to group 1 with endOfBatch=false, so it's cached in org.apache.ignite.raft.jraft.core.NodeImpl.LogEntryAndClosureHandler#tasks and is not processed.
Second event is delivered to group 2 with endOfBatch=true and processed, but first event will remain in queue unprocessed forever, because LogEntryAndClosureHandler are different instances per raft group.
The possible WA for this is to set org.apache.ignite.raft.jraft.option.RaftOptions#applyBatch=1
Reproducible by org.apache.ignite.internal.table.TxDistributedTest_1_1_1#testCrossTable + applyBatch=32 in ignite-15085 branch
.
> Striped Disruptor doesn't work with JRaft event handlers properly
> -----------------------------------------------------------------
>
> Key: IGNITE-15568
> URL: https://issues.apache.org/jira/browse/IGNITE-15568
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexey Scherbakov
> Priority: Major
> Labels: ignite-3
>
> The following scenario is broken:
> # Two raft groups are started and mapped to the same stripe.
> # Two LogEntryAndClosure events are added in quick succession so they form distruptor batch: first for group 1, second for group 2.
> First event is delivered to group 1 with endOfBatch=false, so it's cached in org.apache.ignite.raft.jraft.core.NodeImpl.LogEntryAndClosureHandler#tasks and is not processed.
> Second event is delivered to group 2 with endOfBatch=true and processed, but first event will remain in queue unprocessed forever, because LogEntryAndClosureHandler are different instances per raft group.
> The possible WA for this is to set org.apache.ignite.raft.jraft.option.RaftOptions#applyBatch=1
> Reproducible by org.apache.ignite.internal.table.TxDistributedTest_1_1_1#testCrossTable + applyBatch=32 in ignite-15085 branch
> *Implementation notes*
> My proposal goes bound Disruptor. The striped disruptor implementation has an interceptor that proposes an event to a specific interceptor. Only the last event in the batch has a completion batch flag. For the other RAFT groups, which has been notified in the striped disruptor, required to create an event to fix a batch into the specific group. The new event will be created in the common striped disruptor interceptor, and it will send to a specific interceptor with flag about batch completion.
> The rule of handling the new event is differenced for various interceptor:
> {code:java|title=title=ApplyTaskHandler (FSMCallerImpl#runApplyTask)}
> if (maxCommittedIndex >= 0) {
> doCommitted(maxCommittedIndex);
> return -1;
> }
> {code}
> {code:java|title=LogEntryAndClosureHandler(LogEntryAndClosureHandler#onEvent)}
> if (this.tasks.size() > 0) {
> executeApplyingTasks(this.tasks);
> this.tasks.clear();
> }
> {code}
> {code:java|title=ReadIndexEventHandler(ReadIndexEventHandler#onEvent)}
> if (this.events.size() > 0) {
> executeReadIndexEvents(this.events);
> this.events.clear();
> }
> {code}
> {code:java|title=StableClosureEventHandler(StableClosureEventHandler#onEvent)}
> if (this.ab.size > 0) {
> this.lastId = this.ab.flush();
> setDiskId(this.lastId);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)