You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by "kezhuw (via GitHub)" <gi...@apache.org> on 2025/05/04 12:27:41 UTC
[PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
kezhuw opened a new pull request, #2254:
URL: https://github.com/apache/zookeeper/pull/2254
There are two variants of `ZooKeeperServer::processTxn`. Those two variants diverge significantly since ZOOKEEPER-3484. `processTxn(Request request)` pops outstanding change from `outstandingChanges` and adds txn to `committedLog` for follower to sync in addition to what `processTxn(TxnHeader hdr, Record txn)` does. The `Learner` uses `processTxn(TxnHeader hdr, Record txn)` to commit txn to memory after ZOOKEEPER-4394, which means it leaves `committedLog` untouched in `SYNCHRONIZATION` phase.
This way, a stale follower will have hole in its `committedLog` after joining cluster. The stale follower will propagate the in memory hole to other stale nodes after becoming leader. This causes data loss.
The test case fails on master and 3.9.3, and passes on 3.9.2. So only 3.9.3 is affected.
This commit drops `processTxn(TxnHeader hdr, Record txn)` as `processTxn(Request request)` is capable in `SYNCHRONIZATION` phase too.
Also, this commit rejects discontinuous proposals in `syncWithLeader` and `committedLog`, so to avoid possible data loss.
Refs: ZOOKEEPER-4925, ZOOKEEPER-4394, ZOOKEEPER-3484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#issuecomment-2932370939
The changes for fixing the data loss issue looks good. It's a very critical issue. Thanks for fixing it.
The changes for blocking proposal and commit requests being out of order looks good in general, but please make sure it doesn't introduce any unexpected issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122042468
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZKDatabase.java:
##########
@@ -320,17 +323,30 @@ public void addCommittedProposal(Request request) {
WriteLock wl = logLock.writeLock();
try {
wl.lock();
- if (committedLog.size() > commitLogCount) {
- committedLog.remove();
- minCommittedLog = committedLog.peek().getZxid();
- }
if (committedLog.isEmpty()) {
minCommittedLog = request.zxid;
maxCommittedLog = request.zxid;
+ } else if (request.zxid <= maxCommittedLog) {
+ // This could happen if lastProcessedZxid is rewinded and database is re-synced.
+ // Currently, it only happens in test codes, but it should also be safe for production path.
Review Comment:
Adding a test case for the scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw merged PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121953646
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -780,8 +803,8 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
continue;
}
packetsNotLogged.removeFirst();
- fzk.appendRequest(pif.hdr, pif.rec, pif.digest);
- fzk.processTxn(pif.hdr, pif.rec);
+ fzk.appendRequest(pif.toRequest());
+ fzk.processTxn(pif.toRequest());
Review Comment:
nitpick
How about create the request object and reuse it instead of converting twice?
```
Request request = pif.toRequest()
fzk.appendRequest(request)
fzk.processTxn(request)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121999705
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -630,11 +634,21 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
pif.hdr = logEntry.getHeader();
pif.rec = logEntry.getTxn();
pif.digest = logEntry.getDigest();
- if (pif.hdr.getZxid() != lastQueued + 1) {
- LOG.warn(
- "Got zxid 0x{} expected 0x{}",
+ if (lastQueued == 0) {
+ LOG.info("DIFF sync got first proposal 0x{}", Long.toHexString(pif.hdr.getZxid()));
+ } else if (pif.hdr.getZxid() != lastQueued + 1) {
+ if (ZxidUtils.getEpochFromZxid(pif.hdr.getZxid()) <= ZxidUtils.getEpochFromZxid(lastQueued)) {
+ String msg = String.format("DIFF sync got proposal 0x%s which is not next of last proposal 0x%s",
+ Long.toHexString(pif.hdr.getZxid()),
+ Long.toHexString(lastQueued));
+ LOG.error(msg);
+ throw new Exception(msg);
+ }
+ // We can't tell whether it is a data loss. Given that new epoch is rare,
+ // log at warn should not be too verbose.
+ LOG.warn("DIFF sync got new epoch proposal 0x{}, last proposal 0x{}",
Review Comment:
How about keeping the original WARN statement, so it's very clear what is the actual, what is the expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122561575
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZKDatabase.java:
##########
@@ -320,17 +323,30 @@ public void addCommittedProposal(Request request) {
WriteLock wl = logLock.writeLock();
try {
wl.lock();
- if (committedLog.size() > commitLogCount) {
- committedLog.remove();
- minCommittedLog = committedLog.peek().getZxid();
- }
if (committedLog.isEmpty()) {
minCommittedLog = request.zxid;
maxCommittedLog = request.zxid;
+ } else if (request.zxid <= maxCommittedLog) {
+ // This could happen if lastProcessedZxid is rewinded and database is re-synced.
+ // Currently, it only happens in test codes, but it should also be safe for production path.
Review Comment:
`LoadFromLogTest::testRestore` fails without this. Here is the javadoc of the test.
```java
/**
* Test we can restore the snapshot that has data ahead of the zxid
* of the snapshot file.
*/
```
https://github.com/apache/zookeeper/blob/aeadf574c0463e7e3db092b588033d8347989d4c/zookeeper-server/src/test/java/org/apache/zookeeper/test/LoadFromLogTest.java#L178-L182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122629733
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -630,11 +634,21 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
pif.hdr = logEntry.getHeader();
pif.rec = logEntry.getTxn();
pif.digest = logEntry.getDigest();
- if (pif.hdr.getZxid() != lastQueued + 1) {
- LOG.warn(
- "Got zxid 0x{} expected 0x{}",
+ if (lastQueued == 0) {
+ LOG.info("DIFF sync got first proposal 0x{}", Long.toHexString(pif.hdr.getZxid()));
+ } else if (pif.hdr.getZxid() != lastQueued + 1) {
+ if (ZxidUtils.getEpochFromZxid(pif.hdr.getZxid()) <= ZxidUtils.getEpochFromZxid(lastQueued)) {
+ String msg = String.format("DIFF sync got proposal 0x%s which is not next of last proposal 0x%s",
+ Long.toHexString(pif.hdr.getZxid()),
+ Long.toHexString(lastQueued));
+ LOG.error(msg);
+ throw new Exception(msg);
+ }
+ // We can't tell whether it is a data loss. Given that new epoch is rare,
+ // log at warn should not be too verbose.
+ LOG.warn("DIFF sync got new epoch proposal 0x{}, last proposal 0x{}",
Review Comment:
Add expected zxid in newly introduced method `enforceContinuousProposal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#issuecomment-2942413138
@cnauroth @anmolnar @eolivelli @tisonkun Would you like to take another look ? I plan to merge this in days to not block 3.9.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122040634
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZKDatabase.java:
##########
@@ -320,17 +323,30 @@ public void addCommittedProposal(Request request) {
WriteLock wl = logLock.writeLock();
try {
wl.lock();
- if (committedLog.size() > commitLogCount) {
- committedLog.remove();
- minCommittedLog = committedLog.peek().getZxid();
- }
if (committedLog.isEmpty()) {
minCommittedLog = request.zxid;
maxCommittedLog = request.zxid;
+ } else if (request.zxid <= maxCommittedLog) {
+ // This could happen if lastProcessedZxid is rewinded and database is re-synced.
+ // Currently, it only happens in test codes, but it should also be safe for production path.
+ return;
+ } else if (!allowDiscontinuousProposals
+ && request.zxid != maxCommittedLog + 1
+ && ZxidUtils.getEpochFromZxid(request.zxid) <= ZxidUtils.getEpochFromZxid(maxCommittedLog)) {
+ String msg = String.format(
+ "Committed proposal cached out of order: 0x%s is not the next proposal of 0x%s",
+ ZxidUtils.zxidToString(request.zxid),
+ ZxidUtils.zxidToString(maxCommittedLog));
+ LOG.error(msg);
+ throw new IllegalStateException(msg);
Review Comment:
Do we have a test case for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122631643
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -780,8 +803,8 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
continue;
}
packetsNotLogged.removeFirst();
- fzk.appendRequest(pif.hdr, pif.rec, pif.digest);
- fzk.processTxn(pif.hdr, pif.rec);
+ fzk.appendRequest(pif.toRequest());
+ fzk.processTxn(pif.toRequest());
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2125310908
##########
zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSyncTest.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import java.util.Comparator;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.test.QuorumUtil;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+public class QuorumSyncTest extends ZKTestCase {
+ private QuorumUtil qu;
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (qu != null) {
+ qu.shutdownAll();
+ }
+ }
+
+ @Test
+ public void testStaleDiffSync() throws Exception {
+ qu = new QuorumUtil(2);
+ qu.startAll();
+
+ int[] followerIds = qu.getFollowerQuorumPeers()
+ .stream()
+ .sorted(Comparator.comparingLong(QuorumPeer::getMyId).reversed())
+ .mapToInt(peer -> (int) peer.getMyId()).toArray();
+
+ int follower1 = followerIds[0];
+ int follower2 = followerIds[1];
+
+ String leaderConnectString = qu.getConnectString(qu.getLeaderQuorumPeer());
+ try (ZooKeeper zk = ClientBase.createZKClient(leaderConnectString)) {
+ qu.shutdown(follower2);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/foo" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.shutdown(follower1);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/bar" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.restart(follower1);
+ }
+
+ try (ZooKeeper zk = ClientBase.createZKClient(qu.getConnectionStringForServer(follower1))) {
+ for (int i = 0; i < 10; i++) {
+ String path = "/foo" + i;
+ assertNotNull(zk.exists(path, false), path + " not found");
+ }
+
+ for (int i = 0; i < 10; i++) {
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121953646
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -780,8 +803,8 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
continue;
}
packetsNotLogged.removeFirst();
- fzk.appendRequest(pif.hdr, pif.rec, pif.digest);
- fzk.processTxn(pif.hdr, pif.rec);
+ fzk.appendRequest(pif.toRequest());
+ fzk.processTxn(pif.toRequest());
Review Comment:
nitpick
```suggestion
Request request = pif.toRequest()
fzk.appendRequest(request)
fzk.processTxn(request)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121999705
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -630,11 +634,21 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
pif.hdr = logEntry.getHeader();
pif.rec = logEntry.getTxn();
pif.digest = logEntry.getDigest();
- if (pif.hdr.getZxid() != lastQueued + 1) {
- LOG.warn(
- "Got zxid 0x{} expected 0x{}",
+ if (lastQueued == 0) {
+ LOG.info("DIFF sync got first proposal 0x{}", Long.toHexString(pif.hdr.getZxid()));
+ } else if (pif.hdr.getZxid() != lastQueued + 1) {
+ if (ZxidUtils.getEpochFromZxid(pif.hdr.getZxid()) <= ZxidUtils.getEpochFromZxid(lastQueued)) {
+ String msg = String.format("DIFF sync got proposal 0x%s which is not next of last proposal 0x%s",
+ Long.toHexString(pif.hdr.getZxid()),
+ Long.toHexString(lastQueued));
+ LOG.error(msg);
+ throw new Exception(msg);
+ }
+ // We can't tell whether it is a data loss. Given that new epoch is rare,
+ // log at warn should not be too verbose.
+ LOG.warn("DIFF sync got new epoch proposal 0x{}, last proposal 0x{}",
Review Comment:
How about keeping the original WARN statement, so it's very clear what is the actual and what is the expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122632370
##########
zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSyncTest.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import java.util.Comparator;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.test.QuorumUtil;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+public class QuorumSyncTest extends ZKTestCase {
+ private QuorumUtil qu;
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (qu != null) {
+ qu.shutdownAll();
+ }
+ }
+
+ @Test
+ public void testStaleDiffSync() throws Exception {
+ qu = new QuorumUtil(2);
+ qu.startAll();
+
+ int[] followerIds = qu.getFollowerQuorumPeers()
+ .stream()
+ .sorted(Comparator.comparingLong(QuorumPeer::getMyId).reversed())
+ .mapToInt(peer -> (int) peer.getMyId()).toArray();
+
+ int follower1 = followerIds[0];
+ int follower2 = followerIds[1];
+
+ String leaderConnectString = qu.getConnectString(qu.getLeaderQuorumPeer());
+ try (ZooKeeper zk = ClientBase.createZKClient(leaderConnectString)) {
+ qu.shutdown(follower2);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/foo" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.shutdown(follower1);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/bar" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.restart(follower1);
+ }
+
+ try (ZooKeeper zk = ClientBase.createZKClient(qu.getConnectionStringForServer(follower1))) {
+ for (int i = 0; i < 10; i++) {
+ String path = "/foo" + i;
+ assertNotNull(zk.exists(path, false), path + " not found");
+ }
+
+ for (int i = 0; i < 10; i++) {
Review Comment:
ZOOKEEPER-4925 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122631097
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -696,18 +710,27 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
packet.rec = logEntry.getTxn();
packet.hdr = logEntry.getHeader();
packet.digest = logEntry.getDigest();
- // Log warning message if txn comes out-of-order
- if (packet.hdr.getZxid() != lastQueued + 1) {
- LOG.warn(
- "Got zxid 0x{} expected 0x{}",
- Long.toHexString(packet.hdr.getZxid()),
- Long.toHexString(lastQueued + 1));
+ if (lastQueued == 0) {
+ LOG.info("DIFF sync got first proposal 0x{}", Long.toHexString(packet.hdr.getZxid()));
+ } else if (packet.hdr.getZxid() != lastQueued + 1) {
+ if (ZxidUtils.getEpochFromZxid(packet.hdr.getZxid()) <= ZxidUtils.getEpochFromZxid(lastQueued)) {
+ String msg = String.format("DIFF sync got proposal 0x%s which is not next of last proposal 0x%s",
+ Long.toHexString(packet.hdr.getZxid()),
+ Long.toHexString(lastQueued));
+ LOG.error(msg);
+ throw new Exception(msg);
+ }
+ // We can't tell whether it is a data loss. Given that new epoch is rare,
+ // log at warn should not be too verbose.
+ LOG.warn("DIFF sync got new epoch proposal 0x{}, last proposal 0x{}",
+ Long.toHexString(packet.hdr.getZxid()),
+ Long.toHexString(lastQueued));
Review Comment:
Move them to `enforceContinuousProposal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121976783
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java:
##########
@@ -696,18 +710,27 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception {
packet.rec = logEntry.getTxn();
packet.hdr = logEntry.getHeader();
packet.digest = logEntry.getDigest();
- // Log warning message if txn comes out-of-order
- if (packet.hdr.getZxid() != lastQueued + 1) {
- LOG.warn(
- "Got zxid 0x{} expected 0x{}",
- Long.toHexString(packet.hdr.getZxid()),
- Long.toHexString(lastQueued + 1));
+ if (lastQueued == 0) {
+ LOG.info("DIFF sync got first proposal 0x{}", Long.toHexString(packet.hdr.getZxid()));
+ } else if (packet.hdr.getZxid() != lastQueued + 1) {
+ if (ZxidUtils.getEpochFromZxid(packet.hdr.getZxid()) <= ZxidUtils.getEpochFromZxid(lastQueued)) {
+ String msg = String.format("DIFF sync got proposal 0x%s which is not next of last proposal 0x%s",
+ Long.toHexString(packet.hdr.getZxid()),
+ Long.toHexString(lastQueued));
+ LOG.error(msg);
+ throw new Exception(msg);
+ }
+ // We can't tell whether it is a data loss. Given that new epoch is rare,
+ // log at warn should not be too verbose.
+ LOG.warn("DIFF sync got new epoch proposal 0x{}, last proposal 0x{}",
+ Long.toHexString(packet.hdr.getZxid()),
+ Long.toHexString(lastQueued));
Review Comment:
This is the same code as line 637-651. How about creating a common method to check if proposal is out of order so it can be used in both places?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2121930956
##########
zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSyncTest.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import java.util.Comparator;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.test.QuorumUtil;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+public class QuorumSyncTest extends ZKTestCase {
+ private QuorumUtil qu;
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (qu != null) {
+ qu.shutdownAll();
+ }
+ }
+
+ @Test
+ public void testStaleDiffSync() throws Exception {
+ qu = new QuorumUtil(2);
+ qu.startAll();
+
+ int[] followerIds = qu.getFollowerQuorumPeers()
+ .stream()
+ .sorted(Comparator.comparingLong(QuorumPeer::getMyId).reversed())
+ .mapToInt(peer -> (int) peer.getMyId()).toArray();
+
+ int follower1 = followerIds[0];
+ int follower2 = followerIds[1];
+
+ String leaderConnectString = qu.getConnectString(qu.getLeaderQuorumPeer());
+ try (ZooKeeper zk = ClientBase.createZKClient(leaderConnectString)) {
+ qu.shutdown(follower2);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/foo" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.shutdown(follower1);
+
+ for (int i = 0; i < 10; i++) {
+ zk.create("/bar" + i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ }
+
+ qu.restart(follower1);
+ }
+
+ try (ZooKeeper zk = ClientBase.createZKClient(qu.getConnectionStringForServer(follower1))) {
+ for (int i = 0; i < 10; i++) {
+ String path = "/foo" + i;
+ assertNotNull(zk.exists(path, false), path + " not found");
+ }
+
+ for (int i = 0; i < 10; i++) {
Review Comment:
How about adding the JIRA ticket to help people understand the issue and fix? Basically all the /boo related txns are synced via DIFF and added to the committed logs with the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "li4wang (via GitHub)" <gi...@apache.org>.
li4wang commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122042468
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZKDatabase.java:
##########
@@ -320,17 +323,30 @@ public void addCommittedProposal(Request request) {
WriteLock wl = logLock.writeLock();
try {
wl.lock();
- if (committedLog.size() > commitLogCount) {
- committedLog.remove();
- minCommittedLog = committedLog.peek().getZxid();
- }
if (committedLog.isEmpty()) {
minCommittedLog = request.zxid;
maxCommittedLog = request.zxid;
+ } else if (request.zxid <= maxCommittedLog) {
+ // This could happen if lastProcessedZxid is rewinded and database is re-synced.
+ // Currently, it only happens in test codes, but it should also be safe for production path.
Review Comment:
How about adding a test case for the scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on code in PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#discussion_r2122595721
##########
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZKDatabase.java:
##########
@@ -320,17 +323,30 @@ public void addCommittedProposal(Request request) {
WriteLock wl = logLock.writeLock();
try {
wl.lock();
- if (committedLog.size() > commitLogCount) {
- committedLog.remove();
- minCommittedLog = committedLog.peek().getZxid();
- }
if (committedLog.isEmpty()) {
minCommittedLog = request.zxid;
maxCommittedLog = request.zxid;
+ } else if (request.zxid <= maxCommittedLog) {
+ // This could happen if lastProcessedZxid is rewinded and database is re-synced.
+ // Currently, it only happens in test codes, but it should also be safe for production path.
+ return;
+ } else if (!allowDiscontinuousProposals
+ && request.zxid != maxCommittedLog + 1
+ && ZxidUtils.getEpochFromZxid(request.zxid) <= ZxidUtils.getEpochFromZxid(maxCommittedLog)) {
+ String msg = String.format(
+ "Committed proposal cached out of order: 0x%s is not the next proposal of 0x%s",
+ ZxidUtils.zxidToString(request.zxid),
+ ZxidUtils.zxidToString(maxCommittedLog));
+ LOG.error(msg);
+ throw new IllegalStateException(msg);
Review Comment:
I think it is already covered by existing tests. `allowDiscontinuousProposals`(used by `ZxidRolloverTest`, `TxnLogDigestTest`, both introduce discontinuous proposals on purpose) was introduced to bypass this restriction.
Also this path will fail newly introduced test if we rollback changes involed in sync phase.
```bash
git checkout HEAD^ -- zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/
git checkout HEAD^ -- zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
```
So, I think it is covered, though not for successful tests.
Besides, this is what I state in commit message.
> Also, this commit rejects discontinuous proposals in syncWithLeader and committedLog, so to avoid possible data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4925: Fix data loss due to propagation of discontinuous committedLog [zookeeper]
Posted by "kezhuw (via GitHub)" <gi...@apache.org>.
kezhuw commented on PR #2254:
URL: https://github.com/apache/zookeeper/pull/2254#issuecomment-2933376784
Hi @li4wang, thank for for reviewing this! I have pushed a new commit. Please take another to look and let me know whether it solves your concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org