You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Pratyush Bhatt (Jira)" <ji...@apache.org> on 2024/04/22 19:12:00 UTC
[jira] [Created] (HDDS-10734) [Hbase Ozone] ImportTSV fails during OM Rolling Restart with "SecretManager$InvalidToken: Tampered/Invalid token."
Pratyush Bhatt created HDDS-10734:
-------------------------------------
Summary: [Hbase Ozone] ImportTSV fails during OM Rolling Restart with "SecretManager$InvalidToken: Tampered/Invalid token."
Key: HDDS-10734
URL: https://issues.apache.org/jira/browse/HDDS-10734
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Pratyush Bhatt
Triggering ImportTSV during Rolling restart is failing.
Debugged the issue, and its reproducible everytime when the "reducers" are getting used by ImportTSV and at the same time there is a OM rolling restart stage going on.
{code:java}
2024-04-22 10:16:29,396|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|24/04/22 10:16:29 INFO mapreduce.Job: Task Id : attempt_1713778160624_0007_r_000072_0, Status : FAILED
2024-04-22 10:16:29,434|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|Error: org.apache.hadoop.security.token.SecretManager$InvalidToken: Tampered/Invalid token.
2024-04-22 10:16:29,434|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-04-22 10:16:29,434|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2024-04-22 10:16:29,434|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-04-22 10:16:29,435|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2024-04-22 10:16:29,435|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
2024-04-22 10:16:29,435|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
2024-04-22 10:16:29,435|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:253)
2024-04-22 10:16:29,435|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:115)
2024-04-22 10:16:29,436|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:201)
2024-04-22 10:16:29,436|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
2024-04-22 10:16:29,436|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:111)
2024-04-22 10:16:29,436|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:189)
2024-04-22 10:16:29,436|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3451)
2024-04-22 10:16:29,437|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:161)
2024-04-22 10:16:29,437|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3556)
2024-04-22 10:16:29,437|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3503)
2024-04-22 10:16:29,437|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:521)
2024-04-22 10:16:29,437|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:269)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:173)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at java.security.AccessController.doPrivileged(Native Method)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at javax.security.auth.Subject.doAs(Subject.java:422)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
2024-04-22 10:16:29,438|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Tampered/Invalid token.
2024-04-22 10:16:29,439|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1616)
2024-04-22 10:16:29,439|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.Client.call(Client.java:1562)
2024-04-22 10:16:29,439|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.Client.call(Client.java:1459)
2024-04-22 10:16:29,439|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
2024-04-22 10:16:29,439|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
2024-04-22 10:16:29,440|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at com.sun.proxy.$Proxy17.submitRequest(Unknown Source)
2024-04-22 10:16:29,440|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
2024-04-22 10:16:29,440|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-04-22 10:16:29,440|INFO|Thread-37|machine.py:205 - run()||GUID=51e988c6-6805-43d1-9290-eb6f667ac2dd|at java.lang.reflect.Method.invoke(Method.java:498) {code}
Checked the leader OM logs, shows below:
{code:java}
2024-04-22 10:16:24,671 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.133.64:46032:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.)
2024-04-22 10:16:24,671 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.68.1:43592:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.)
2024-04-22 10:16:24,672 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.170.2:41974:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.)
2024-04-22 10:16:24,672 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.133.64:46020:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.)
2024-04-22 10:16:24,672 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.11.131:50274:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.)
2024-04-22 10:16:24,675 WARN [Socket Reader #1 for port 9862]-SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.140.11.131:50290:null (DIGEST-MD5: IO error acquiring password) with true cause: (OM:om102 is not the leader. Could not determine the leader node.) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org