You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "migesok (via GitHub)" <gi...@apache.org> on 2025/06/02 11:04:07 UTC
[D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
GitHub user migesok created a discussion: Why AsyncWriteJournal.Resequencer is needed?
Hi!
During an investigation of a latency issue in our code using classic persistent actors, we found that the "lost" time is spent waiting inside `AsyncWriteJournal.Resequencer`. As I understood, it reorders journal write responses to be in the same order as the requests. The order is global (per plugin instance). Our problem was gone when we patched the journal plugin logic to remove `Resequencer`. At the same time no other issues arose during tests.
Why this ordering is needed? Why it has to be global and not, say, per persistence ID? Per actor instance ID?
What I found so far myself:
- no additional pekko-persistence tests fail even if I remove the resequencer (though some test fail for me even without changes because of the outdated leveldb JNI library)
- that piece of code is almost as old as the Akka itself - it was there in the first Akka Persistence commits:
https://github.com/akka/akka/pull/1751
I also found that the logic was added to the original `eventsourced` project (from which it was ported to Akka) when it added support for "async" journal plugins (not sure what that means 100%):
https://github.com/krasserm/eventsourced/commit/f200b123c60917224deffbbd4c7bfd37ac59d71b
- journal resequencing logic is mentioned in defer and deferAsync implementations so I assume they are related somehow
GitHub link: https://github.com/apache/pekko/discussions/1881
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
GitHub user pjfanning edited a comment on the discussion: Why AsyncWriteJournal.Resequencer is needed?
I googled around and the only real discussion about Akka/Pekko Resequencers that I could find was in this book.
https://github.com/yyccR/papers/blob/master/genetic%20algorithm/Reactive.Messaging.Patterns.with.the.Actor.Model.Applications.and.Integration.in.Scala.and.Akka.pdf
Mainly about Actors but it does discuss Persistent Actors too.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13345703
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "migesok (via GitHub)" <gi...@apache.org>.
GitHub user migesok added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
@pjfanning I ran tests in the `persistence-typed` module, it worked. Btw suggestions how to circumvent leveldbjni issue are very welcome. I have an m1 mac and I think the library just doesn't have the native code for mac os aarch64. I don't think currently there is a way to ran pekko project tests green 100% on an m1 mac because of that.
Currently I'm trying to come up with a test where I can prove that absence of journal plugin response reordering breaks persistAsync, defer or deferAsync. Or doesn't.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13343173
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "ptrdom (via GitHub)" <gi...@apache.org>.
GitHub user ptrdom edited a comment on the discussion: Why AsyncWriteJournal.Resequencer is needed?
I think we cannot remove this ordering behaviour either way, no matter if no library-level code or test breaks. As suggested, best we can do is add a config to disable.
IMHO this ordering behaviour is only relevant to very funky classic persistent actor implementations, for any new typed persistence code this ordering behaviour is only a performance penalty, so maybe we could disable it by default in some future major release.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344598
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "migesok (via GitHub)" <gi...@apache.org>.
GitHub user migesok added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
"very funky classic persistent actor implementations" - that's what I'm dealing with, I believe. We have persistAsync and deferAsync in ungodly combinations. But I can't make it fail in a test even without resequencer which makes me think there is some other built-in mechanism which makes it work. Or I don't understand the type of failures resequencer was made to avert and I'm yet to hit it.
My current assumption was that resequencer was meant to keep the order of async callbacks preserved (persist and defer) but they seem to be in order even without it. Even when I deliberately reorder journal actor responses to be in a wrong order.
We already have the fix removing resequencer applied locally for some of our projects. I'm trying to understand whether it is safe to extend it company-wise and whether it should be contributed to Pekko. For this reason I'm trying to understand the consequences of removing resequencer.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344780
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "ptrdom (via GitHub)" <gi...@apache.org>.
GitHub user ptrdom added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
I think we cannot remove this ordering behaviour either way, not matter if no library-level code or test breaks. As suggested, best we can do is add a config to disable.
IMHO this ordering behaviour is only relevant to very funky classic persistent actor implementations, for any new typed persistence code this ordering behaviour is only a performance penalty, so maybe we could disable it by default in some future major release.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344598
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "ptrdom (via GitHub)" <gi...@apache.org>.
GitHub user ptrdom added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
I think this is what can happen if there is no resequencer:
1) Actor defers two writes and then persists them.
2) No resequencer, so second write might complete first, and first write can compete second or fail:
3) Handlers are executed in order, but writes are completed out of order, so you are executing first write's handler on completion second write.
So if user-level code is sensitive to this behaviour, which I would call a bug, then it can break. The way I see it this is more relevant to failure cases - using the above example, first write might not even complete, but its handler is being triggered by the second write succeeding.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13345026
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "migesok (via GitHub)" <gi...@apache.org>.
GitHub user migesok edited a comment on the discussion: Why AsyncWriteJournal.Resequencer is needed?
"very funky classic persistent actor implementations" - that's what I'm dealing with, I believe. We have persistAsync and deferAsync in ungodly combinations. But I can't make it fail in a test even without resequencer which makes me think there is some other built-in mechanism which makes it work. Or I don't understand the type of failures resequencer was made to avert and I'm yet to hit it.
My current assumption is that resequencer was meant to keep the order of async callbacks preserved (persist and defer) but they seem to be in order even without it. Even when I deliberately reorder journal actor responses to be in a wrong order.
We already have the fix removing resequencer applied locally for some of our projects. I'm trying to understand whether it is safe to extend it company-wise and whether it should be contributed to Pekko. For this reason I'm trying to understand the consequences of removing resequencer.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344780
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
GitHub user pjfanning added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
I googled around and the only real discussion about Akka/Pekko Resequencers that I could find was in this book.
https://github.com/yyccR/papers/blob/master/genetic%20algorithm/Reactive.Messaging.Patterns.with.the.Actor.Model.Applications.and.Integration.in.Scala.and.Akka.pdf
Mainly about Actors but it does not discuss Persistent Actors too.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13345703
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "migesok (via GitHub)" <gi...@apache.org>.
GitHub user migesok added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
After some experiments and diving into Akka 2.6 and Pekko code, I came to this:
I'm almost certain that neither for a classic persistent actor, nor for a typed, one can observe differences in behaviour between AsyncWriteJournal with and without resequencer. It is because
- persistent actors inside 1 "incarnation" (between restarts) never do 2 asyncWriteMessages calls simultaneously, they are always done 1 after another
- persistent actors discard and ignore asyncWriteMessages result responses from requests initiated by different "incarnations" of the same actor (courtesy of `actorInstanceId`)
So the only way to observe differences in behaviour with and without resequencer is to stage an interplay between 2 persistent actors, giving them commands to persist events, and observing order in which replies come back. It should never be visible if you look at isolated actors.
On one hand, resequencer can be dropped altogether, on another - could there be systems around which really depend on such intricate ordering guarantees which manifests only when several persistent actors work together? Possible but unlikely.
To be on the safe side, resequencer switch could be a plugin level config option.
My current plan is to roll out the change first inside our codebase. We are still running mostly on Akka 2.6 with Pekko migration in progress, using in-house journal and snapshot plugins. Then we will try to backport it to Pekko, if maintainers agree to take it.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13367927
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "migesok (via GitHub)" <gi...@apache.org>.
GitHub user migesok added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
@ptrdom can you help me a bit with translating this into PersistentActor logic? By the first point you mean calling deferAsync twice and then doing persistAsync inside their respective deferAsync callbacks?
I made a small "chaos" test harness for this. If I can put it into a chain of persist and defer calls I can try to confirm it.
So far I wasn't able to make PersistentActor run 2 actual journal `asyncWriteMessages` operations in parallel. It seems to collect all outstanding writes and dump then in 1 call, no matter what I do. Probably I just don't understand it that well.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13345303
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
GitHub user pjfanning added a comment to the discussion: Why AsyncWriteJournal.Resequencer is needed?
Thanks @migesok for raising this.
I'm not aware of why this Resequencer is used. If, as you suggest, the resequencer doesn't seem to affect the results of any tests, maybe, we could add a config setting that allows it to be skipped. My preference, for backward compatibility, is that the Resequencer is used by default. It might be best not to do anything until other contributors have time to provide their opinions or context.
Have you tried running any of the tests in modules like pekko-persistence-jdbc?
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13343107
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "ptrdom (via GitHub)" <gi...@apache.org>.
GitHub user ptrdom edited a comment on the discussion: Why AsyncWriteJournal.Resequencer is needed?
I think we cannot remove this ordering behaviour either way, no matter if no library-level code or test breaks. As suggested, best we can do is add a config to disable.
IMHO this ordering behaviour is only relevant for very funky classic persistent actor implementations, for any new typed persistence code this ordering behaviour is only a performance penalty, so maybe we could disable it by default in some future major release.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344598
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [D] Why AsyncWriteJournal.Resequencer is needed? [pekko]
Posted by "ptrdom (via GitHub)" <gi...@apache.org>.
GitHub user ptrdom edited a comment on the discussion: Why AsyncWriteJournal.Resequencer is needed?
I think we cannot remove this ordering behaviour either way, no matter if no library-level code or test breaks. As suggested, best we can do is add a config to disable.
IMHO this ordering behaviour is only relevant to very funky classic persistent actor implementations, for any new typed persistence code this ordering behaviour is only a performance penalty, so maybe we could disable it by default in some future major release.
GitHub link: https://github.com/apache/pekko/discussions/1881#discussioncomment-13344598
----
This is an automatically sent email for notifications@pekko.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@pekko.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org