You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Stefan Egli (Jira)" <ji...@apache.org> on 2019/09/16 13:03:00 UTC
[jira] [Commented] (OAK-8627) Avoid late-arriving lastRev update
from crashed instance
[ https://issues.apache.org/jira/browse/OAK-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930518#comment-16930518 ]
Stefan Egli commented on OAK-8627:
----------------------------------
Pushed changes in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-8627] :
* [this 1st commit|https://github.com/stefan-egli/jackrabbit-oak/commit/f9c5ece53099714ee584986e75c4c7e8470409ac] is originally from [~mreutegg]
* [this 2nd commit|https://github.com/stefan-egli/jackrabbit-oak/commit/7e8ac4c0bf76db824a051c304a03ce02b93a40a1] fixes the test failures: basically some tests where doing a {{recover}} and thereafter in tearDown a {{dispose}} which failed due to the last backgroundWrite not succeeding (due to the recover). This is now an expected situation, hence I've adjusted the affected test cases to accept this. Another case was a concurrent recover which now can fail due to the new check : also handling that as acceptable now.
Other than that what I see as remaining failure scenario is that the root update could succeed but the lastWrittenRootRev not updated (eg due to a crash) - but this was actually already handled correctly now : the DocumentDiscoveryLiteService has a [{{lastKnownRevision >= lastWrittenRootRev}}|https://github.com/apache/jackrabbit-oak/blob/637a58a1c185033c6bfc2ca8259662fc1ad68255/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentDiscoveryLiteService.java#L533] check, so it handles this fine.
I'd suggest to apply this change. [~mreutegg] as you originally started working on this, can you please review and approve the change? thx!
> Avoid late-arriving lastRev update from crashed instance
> --------------------------------------------------------
>
> Key: OAK-8627
> URL: https://issues.apache.org/jira/browse/OAK-8627
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
> Affects Versions: 1.16.0
> Reporter: Stefan Egli
> Assignee: Stefan Egli
> Priority: Major
>
> Recently a deployment with a two node cluster showed a Sling Discovery Oak with a cluster view that had a clusterId stuck in the deactivating state.
> According to the entry in the clusterNodes collection, the clusterId in the deactivating state was inactive. However, the revisions for the _lastRev entry on the root document and the lastWrittenRootRev did not match. The latter was slightly more recent. This caused the Sling Discovery Oak to consider the clusterId as not entirely shut down.
> While there is no direct proof, one theoretical scenario [~mreutegg] identified as a _potential_ root cause was that it can happen that the lastRev for a clusterId on the root document is set back to an earlier value due to a race condition:
> Before the lease expiry, the backgorund update thread could have issued an update for the root document, which then took a very long time to reach the DocumentStore, longer than the lease timeout and recovery which must have been done by another instance meanwhile.
> If such a late-arriving update of the {{_lastRev}} is possible, then the reset of the lastRev value on the root document could be explained, since the update is currently done unconditionally.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)