You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by GitBox <gi...@apache.org> on 2022/01/06 15:51:49 UTC
[GitHub] [jackrabbit-oak] mreutegg opened a new pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
mreutegg opened a new pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453
Test to reproduce problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#issuecomment-1007301406
@Joscorbe can you please review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg merged pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg merged pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781317051
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.apache.jackrabbit.oak.plugins.document.Collection.CLUSTER_NODES;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ fail("recovery must fail with expired lease");
Review comment:
And the test will fail if the two calls above do not throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781311716
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.apache.jackrabbit.oak.plugins.document.Collection.CLUSTER_NODES;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ } catch (DocumentStoreException e) {
+ // this is expected. any of the above LastRevRecoveryAgent calls
+ // can and should fail because the lease for ns2 expired
Review comment:
> does that mean both lines in the try/catch block can fail?
At least one of the calls should fail. Without the fix `agent.performRecoveryIfNeeded()` fails with an exception. I didn't want the test to prescribe where the implementation should fail. But with the current fix no exception is actually thrown and the test does not detect it.
I'll change both the test to fail when no exception is thrown and update LastRevRecoveryAgent.recover() to throw an exception when it detects its own lease timed out. Let me know what you think after that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] stefan-egli commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
stefan-egli commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781399971
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,28 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ ClusterNodeInfoDocument me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ if (me != null && me.isRecoveryNeeded(now)) {
+ String msg = String.format(
+ "Own clusterId %s has a leaseEnd %s (%s) older than current time %s (%s). " +
+ "Refusing to run recovery on clusterId %s.",
+ revisionContext.getClusterId(), me.getLeaseEndTime(),
+ asISO8601(me.getLeaseEndTime()), now, asISO8601(now),
+ clusterId);
+ throw new DocumentStoreException(msg);
Review comment:
ack, thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#issuecomment-1007301406
@Joscorbe can you please review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781397017
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,28 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ ClusterNodeInfoDocument me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ if (me != null && me.isRecoveryNeeded(now)) {
+ String msg = String.format(
+ "Own clusterId %s has a leaseEnd %s (%s) older than current time %s (%s). " +
+ "Refusing to run recovery on clusterId %s.",
+ revisionContext.getClusterId(), me.getLeaseEndTime(),
+ asISO8601(me.getLeaseEndTime()), now, asISO8601(now),
+ clusterId);
+ throw new DocumentStoreException(msg);
Review comment:
The code path hitting this potential exception is via LastRevRecoveryJob.run().
https://github.com/apache/jackrabbit-oak/blob/954d62c5543b97ac9da51ac670e7b54cc09ab5ca/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java#L1011
It does not handle the exception, but I think that's fine. The job is scheduled at a fixed rate here:
https://github.com/apache/jackrabbit-oak/blob/954d62c5543b97ac9da51ac670e7b54cc09ab5ca/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java#L822
It would just run again when it's time, but more likely the lease mechanism shuts down the service before this happens as a reaction to the expired lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r780950882
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ } catch (DocumentStoreException e) {
+ // this is expected. any of the above LastRevRecoveryAgent calls
+ // can and should fail because the lease for ns2 expired
+ }
+
+ ClusterNodeInfoDocument infoDoc = store.find(Collection.CLUSTER_NODES, "" + 1);
Review comment:
Yes, you are right. Don't know what I was thinking here. Probably copied the line and adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781315064
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,28 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ ClusterNodeInfoDocument me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ if (me != null && me.isRecoveryNeeded(now)) {
+ String msg = String.format(
+ "Own clusterId %s has a leaseEnd %s (%s) older than current time %s (%s). " +
+ "Refusing to run recovery on clusterId %s.",
+ revisionContext.getClusterId(), me.getLeaseEndTime(),
+ asISO8601(me.getLeaseEndTime()), now, asISO8601(now),
+ clusterId);
+ throw new DocumentStoreException(msg);
Review comment:
@stefan-egli now the method throws an exception instead of returning zero. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] Joscorbe commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
Joscorbe commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r780417034
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ } catch (DocumentStoreException e) {
+ // this is expected. any of the above LastRevRecoveryAgent calls
+ // can and should fail because the lease for ns2 expired
+ }
+
+ ClusterNodeInfoDocument infoDoc = store.find(Collection.CLUSTER_NODES, "" + 1);
Review comment:
Can't we just write `"1"` instead of `"" + 1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] Joscorbe commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
Joscorbe commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r780417034
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ } catch (DocumentStoreException e) {
+ // this is expected. any of the above LastRevRecoveryAgent calls
+ // can and should fail because the lease for ns2 expired
+ }
+
+ ClusterNodeInfoDocument infoDoc = store.find(Collection.CLUSTER_NODES, "" + 1);
Review comment:
Can't we just write `"1"` instead of `"" + 1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781256843
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,29 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ ClusterNodeInfoDocument me = null;
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ }
+ if (me != null && me.isRecoveryNeeded(now)) {
Review comment:
Good suggestion. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] stefan-egli commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
stefan-egli commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781049236
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,29 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ ClusterNodeInfoDocument me = null;
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ }
+ if (me != null && me.isRecoveryNeeded(now)) {
Review comment:
Since `me` is only used within the above
```if (clusterId != revisionContext.getClusterId()) {```
block, this block here could be moved 1 line up into that block. That could make it slightly easier to read - since the scope it then smaller. wdyt?
##########
File path: oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/RecoveryWithIncorrectClockTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.jackrabbit.oak.plugins.document.memory.MemoryDocumentStore;
+import org.apache.jackrabbit.oak.stats.Clock;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static org.apache.jackrabbit.oak.plugins.document.Collection.CLUSTER_NODES;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RecoveryWithIncorrectClockTest {
+
+ @Rule
+ public DocumentMKBuilderProvider builderProvider = new DocumentMKBuilderProvider();
+
+ private final DocumentStore store = new MemoryDocumentStore();
+ private final FailingDocumentStore s1 = new FailingDocumentStore(store);
+
+ private final Clock clock = new Clock.Virtual();
+
+ private DocumentNodeStore ns1;
+
+ private DocumentNodeStore ns2;
+
+ @Before
+ public void before() throws Exception {
+ clock.waitUntil(System.currentTimeMillis());
+ Revision.setClock(clock);
+ ClusterNodeInfo.setClock(clock);
+ ns1 = createDocumentNodeStore(1, s1);
+ ns2 = createDocumentNodeStore(2, store);
+ }
+
+ @AfterClass
+ public static void after() {
+ Revision.resetClockToDefault();
+ ClusterNodeInfo.resetClockToDefault();
+ }
+
+ @Test
+ public void recover() throws Exception {
+ // prevent ns1 from writing anything. This will also prevent any
+ // further updates to clusterNodes entry 1
+ s1.fail().after(0).eternally();
+
+ long now = clock.getTime();
+ // simulate a sudden jump of time to the system clock
+ clock.waitUntil(now + TimeUnit.HOURS.toMillis(1));
+ long ahead = clock.getTime();
+ // ns2 now sees an expired lease for clusterId 1
+ LastRevRecoveryAgent agent = ns2.getLastRevRecoveryAgent();
+ try {
+ assertTrue(agent.isRecoveryNeeded());
+ agent.performRecoveryIfNeeded();
+ } catch (DocumentStoreException e) {
+ // this is expected. any of the above LastRevRecoveryAgent calls
+ // can and should fail because the lease for ns2 expired
Review comment:
does that mean both lines in the `try/catch` block can fail? in that case, wouldn't it make sense to put each one in an individual `try/catch` block (as otherwise you're not testing that both lines fail)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] stefan-egli commented on a change in pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
stefan-egli commented on a change in pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#discussion_r781368029
##########
File path: oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
##########
@@ -142,9 +143,28 @@ public int recover(int clusterId, long waitUntil)
ClusterNodeInfoDocument nodeInfo = missingLastRevUtil.getClusterNodeInfo(clusterId);
if (nodeInfo != null) {
+ // Check our own lease before running recovery for another
+ // clusterId (OAK-9656)
+ long now = revisionContext.getClock().getTime();
+ if (clusterId != revisionContext.getClusterId()) {
+ // Get leaseEnd from our own cluster node info, unless
+ // we are doing recovery on startup for the clusterId
+ // we want to acquire. Then it's fine to go ahead with
+ // an expired lease.
+ ClusterNodeInfoDocument me = missingLastRevUtil.getClusterNodeInfo(revisionContext.getClusterId());
+ if (me != null && me.isRecoveryNeeded(now)) {
+ String msg = String.format(
+ "Own clusterId %s has a leaseEnd %s (%s) older than current time %s (%s). " +
+ "Refusing to run recovery on clusterId %s.",
+ revisionContext.getClusterId(), me.getLeaseEndTime(),
+ asISO8601(me.getLeaseEndTime()), now, asISO8601(now),
+ clusterId);
+ throw new DocumentStoreException(msg);
Review comment:
ok, I'm assuming callers will be able to handle this exception? in that case this indeed seems better (as it should be a rather unfrequent case, so an exception seems appropriate)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] stefan-egli commented on pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
stefan-egli commented on pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#issuecomment-1009112874
Thx for the explanations, makes sense now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mreutegg commented on pull request #453: OAK-9656: Recovery runs mistakenly when system clock jumps ahead
Posted by GitBox <gi...@apache.org>.
mreutegg commented on pull request #453:
URL: https://github.com/apache/jackrabbit-oak/pull/453#issuecomment-1008998035
> What I don't understand is whether it fails ("eventually notices") in the same run or the next.
At some point while running recovery the process may notice that its lease expired and fail the recovery with an exception. This may happen when the recovery reads or updates documents and the `LeaseCheckDocumentStoreWrapper` refuses to perform the operation.
> Also since the 2nd sentence states that it fails on the "next lease update" - but unclear whether that's triggered by the time-jumped node or by any other one.
That's indeed confusing. The second sentence is about the impact on the node that has the correct time. The node with the incorrect time wants to run recovery on the clusterId of the node with the correct time. At this point the node with the correct time will observe its entry in the clusterNodes collection updated by someone else and conclude that its lease must have expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org