You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by "ztzg (via GitHub)" <gi...@apache.org> on 2024/08/28 12:10:35 UTC
[PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
ztzg opened a new pull request, #2183:
URL: https://github.com/apache/zookeeper/pull/2183
ZooKeeper snapshots are *fuzzy*, as the server does not stop processing requests while ACLs and nodes are being streamed to disk.
ACLs, notably, are streamed *first*, as a mapping between the full serialized ACL and an "ACL ID" referenced by the node.
Consequently, a snapshot can very well contain ACL IDs which do not exist in the mapping. Prior to ZOOKEEPER-4799, such situations would produce harmless (if annoying) "Ignoring acl XYZ as it does not exist in the cache" INFO entries in the server logs.
With ZOOKEEPER-4799, we started "eagerly" fetching the referenced ACLs in `DataTree` operations such as `createNode`, `deleteNode`, etc.—as opposed to just fetching them from request processors.
This can result in fatal errors during the `fastForwardFromEdits` phase of restoring a database, when transactions are processed on top of an inconsistent data tree—preventing the server from starting.
The errors are thrown in this code path:
``` java
// ReferenceCountedACLCache.java:90
List<ACL> acls = longKeyMap.get(longVal);
if (acls == null) {
LOG.error("ERROR: ACL not available for long {}", longVal);
throw new RuntimeException("Failed to fetch acls for " + longVal);
}
```
Here is a scenario leading to such a failure:
- An existing node `/foo`, sporting an unique ACL, is deleted. This is recorded in transaction log `$SNAP-1`; said ACL is also deallocated;
- Snapshot `$SNAP` is started;
- The ACL map is serialized to `$SNAP`;
- A new node `/foo` sporting the same unique ACL is created in a portion of the data tree which still has to be serialized;
- Node `/foo` is serialized to `$SNAP`—but its ACL isn't;
- The server is restarted;
- The `DataTree` is initialized from `$SNAP`, including node `/foo` with a dangling ACL reference;
- Transaction log `$SNAP-1` is being replayed, leading to a `deleteNode("/foo")`;
- `getACL(node)` panics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar closed pull request #2183: ZOOKEEPER-4846: Failure to reload database due to missing ACL
URL: https://github.com/apache/zookeeper/pull/2183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #2183:
URL: https://github.com/apache/zookeeper/pull/2183#issuecomment-2651428917
Superseded by #2222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #2183:
URL: https://github.com/apache/zookeeper/pull/2183#issuecomment-2641341655
I've found the soluton. PTAL my comment in the Jira:
https://issues.apache.org/jira/browse/ZOOKEEPER-4846?focusedCommentId=17924725&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17924725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #2183:
URL: https://github.com/apache/zookeeper/pull/2183#issuecomment-2638033144
hi @ztzg , are u still working on this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "ztzg (via GitHub)" <gi...@apache.org>.
ztzg commented on PR #2183:
URL: https://github.com/apache/zookeeper/pull/2183#issuecomment-2639396703
Hi @anmolnar,
> hi @ztzg , are u still working on this patch?
We have been using it for a while without adverse effects. I am not working on it, but some decisions in the patch are perhaps a bit controversial. I will "undraft" if it helps getting more reviews & comments :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ZOOKEEPER-4846: Failure to reload database due to missing ACL [zookeeper]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #2183:
URL: https://github.com/apache/zookeeper/pull/2183#issuecomment-2644155384
Apols @ztzg , I quickly created the alternate patch to get feedback as soon as possible: #2222
Don't wanna take the credit, so we should commit as yours eventually once we have agreement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org