You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/09/23 18:39:25 UTC
[GitHub] [accumulo] milleruntime opened a new issue, #2955: FATAL Error performing upgrade validating ZNode ACLs
milleruntime opened a new issue, #2955:
URL: https://github.com/apache/accumulo/issues/2955
I installed 2.0.1 using Uno, manually inserted some data and then shutdown Accumulo. I then changed my uno config to point to snapshot and did a `uno install accumulo --no-deps`. I got this error after starting Accumulo:
<pre>
2022-09-23T14:36:16,822 [upgrade.UpgradeCoordinator] ERROR: FATAL: Error performing upgrade
java.lang.RuntimeException: Upgrade Failed! Error validating ZNode ACLs. Check the log for specific failed paths, check ZooKeeper troubleshooting in user documentation for instructions on how to fix.
at org.apache.accumulo.manager.upgrade.Upgrader9to10.validateACLs(Upgrader9to10.java:171) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.manager.upgrade.Upgrader9to10.upgradeZookeeper(Upgrader9to10.java:135) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.manager.upgrade.UpgradeCoordinator.upgradeZookeeper(UpgradeCoordinator.java:157) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.manager.Manager.setManagerState(Manager.java:263) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.manager.Manager.getManagerLock(Manager.java:1493) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.manager.Manager.run(Manager.java:1039) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1256550696
Can you check the ACL in ZooKeeper with the zkCli and post what they are? It would also be interesting which nodes have the unexpected ACLs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1258080706
I see it in the expanded section now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1258077388
The upgrade code prints out the offending nodes in the log [here](https://github.com/apache/accumulo/blob/main/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java#L160)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1256725238
I installed 2.0.1, and created tables and then shutdown and installed 2.1-SNAPSHOT without problems.
```
Under 2.0.1:
Shell - Apache Accumulo Interactive Shell
-
- version: 2.0.1
- instance name: uno
- instance id: 32494461-ccb1-443c-9724-ccc94fcc0292
-
- type 'help' for a list of available commands
-
root@uno> tables -ls
accumulo.metadata => !0
accumulo.root => +r
accumulo.replication => +rep
trace => 1
tbl1 => 2
ns1.tbl2 => 4
ns1.tbl3 => 5
ns2.tbl4 => 7
ns2.tbl5 =>
Then: 8
uno stop accumulo
-- edit config and resource new env
uno install accumulo --no-deps
uno start accumulo
After the upgrade:
Shell - Apache Accumulo Interactive Shell
-
- version: 2.1.0-SNAPSHOT
- instance name: uno
- instance id: 32494461-ccb1-443c-9724-ccc94fcc0292
-
- type 'help' for a list of available commands
-
root@uno> tables -ls
accumulo.metadata => !0
accumulo.root => +r
accumulo.replication => +rep
trace => 1
tbl1 => 2
ns1.tbl2 => 4
ns1.tbl3 => 5
ns2.tbl4 => 7
ns2.tbl5 => 8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1258208208
We could do something like the following.
```
diff --git a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
index 96315e08ba..d4917ee6f1 100644
--- a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
+++ b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
@@ -72,7 +72,6 @@ import org.apache.accumulo.core.spi.compaction.SimpleCompactionDispatcher;
import org.apache.accumulo.core.tabletserver.log.LogEntry;
import org.apache.accumulo.core.util.HostAndPort;
import org.apache.accumulo.fate.zookeeper.ZooReaderWriter;
-import org.apache.accumulo.fate.zookeeper.ZooUtil;
import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeExistsPolicy;
import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeMissingPolicy;
import org.apache.accumulo.server.ServerContext;
@@ -91,8 +90,11 @@ import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.KeeperException.NoNodeException;
import org.apache.zookeeper.ZKUtil;
import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.ZooDefs.Perms;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -147,6 +149,17 @@ public class Upgrader9to10 implements Upgrader {
final ZooReaderWriter zrw = context.getZooReaderWriter();
final ZooKeeper zk = zrw.getZooKeeper();
final String rootPath = context.getZooKeeperRoot();
+
+ String secret = context.getConfiguration().get(Property.INSTANCE_SECRET);
+ final String scheme = "digest";
+ final String auth = ("accumulo:" + secret);
+ final Id instance_auth = new Id(scheme, auth);
+ final List<ACL> privateWithAuth = new ArrayList<>();
+ privateWithAuth.add(new ACL(Perms.ALL, instance_auth));
+ final List<ACL> publicWithAuth = new ArrayList<>();
+ publicWithAuth.addAll(privateWithAuth);
+ publicWithAuth.add(new ACL(Perms.READ, Ids.ANYONE_ID_UNSAFE));
+
try {
ZKUtil.visitSubTreeDFS(zk, rootPath, false, (rc, path, ctx, name) -> {
try {
@@ -155,7 +168,7 @@ public class Upgrader9to10 implements Upgrader {
if (((path.equals(Constants.ZROOT) || path.equals(Constants.ZROOT + Constants.ZINSTANCES))
&& !acls.equals(ZooDefs.Ids.OPEN_ACL_UNSAFE))
- || (!ZooUtil.PRIVATE.equals(acls) && !ZooUtil.PUBLIC.equals(acls))) {
+ || (!privateWithAuth.equals(acls) && !publicWithAuth.equals(acls))) {
log.error("ZNode at {} has unexpected ACL: {}", path, acls);
aclErrorOccurred.set(true);
} else {
```
In talking with @EdColeman - he wondered if this would put us into a corner such that only digest auth is supported for the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2955:
URL: https://github.com/apache/accumulo/issues/2955#issuecomment-1256815722
Additional info - with set and expected ACLs
<details>
<summary>acl info before and after upgrade - click to expand</summary>
```
Clean install 2.0.1:
Shell - Apache Accumulo Interactive Shell
-
- version: 2.0.1
- instance name: uno
- instance id: 05fb2a97-3f82-4666-a660-238f0267082d
-
- type 'help' for a list of available commands
-
root@uno> createnamespace ns1
root@uno> createtable tbl1
root@uno tbl1> createtable ns1.tbl2
root@uno ns1.tbl2> tables -ls
accumulo.metadata => !0
accumulo.root => +r
accumulo.replication => +rep
trace => 1
tbl1 => 3
ns1.tbl2 => 4
In ZooKeeper:
[zk: localhost:2181(CONNECTED) 0] addauth digest accumulo:uno
[zk: localhost:2181(CONNECTED) 1] getAcl /accumulo/05fb2a97-3f82-4666-a660-238f0267082d/tables/3
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
[zk: localhost:2181(CONNECTED) 2] getAcl /accumulo/05fb2a97-3f82-4666-a660-238f0267082d/namespaces/2
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
After upgrade:
Errors in manager log
2022-09-24T00:20:02,985 [upgrade.Upgrader9to10] ERROR: ZNode at /accumulo/d1097784-fd9c-4b6e-baa5-3933c579be06/users/root/Tables/1 has unexpected ACL: [31,s{'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
}
, 1,s{'world,'anyone}
]
[zk: localhost:2181(CONNECTED) 4] getAcl /accumulo/05fb2a97-3f82-4666-a660-238f0267082d/tables/3
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
[zk: localhost:2181(CONNECTED) 5] getAcl /accumulo/05fb2a97-3f82-4666-a660-238f0267082d/namespaces/2
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
On 2.1.0-SNAPSHOT clean install:
Shell - Apache Accumulo Interactive Shell
-
- version: 2.1.0-SNAPSHOT
- instance name: uno
- instance id: c14a4798-7f33-40be-af8b-f5cc2f4592c0
-
- type 'help' for a list of available commands
-
root@uno> createnamespace ns1
root@uno> createtable tbl1
root@uno tbl1> tables -ls
accumulo.metadata => !0
accumulo.root => +r
accumulo.replication => +rep
tbl1 => 2
[zk: localhost:2181(CONNECTED) 2] getAcl /accumulo/c14a4798-7f33-40be-af8b-f5cc2f4592c0/tables/2
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
[zk: localhost:2181(CONNECTED) 3] getAcl /accumulo/c14a4798-7f33-40be-af8b-f5cc2f4592c0/namespaces/1
'digest,'accumulo:Xndt9++GZKIalorKghvMJTX3z14=
: cdrwa
'world,'anyone
: r
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman closed issue #2955: FATAL Error performing upgrade validating ZNode ACLs
Posted by GitBox <gi...@apache.org>.
EdColeman closed issue #2955: FATAL Error performing upgrade validating ZNode ACLs
URL: https://github.com/apache/accumulo/issues/2955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org