You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2022/05/18 20:01:00 UTC
[jira] [Comment Edited] (HDDS-6763) SCM crashed while getting certificate from SCMCertStore
[ https://issues.apache.org/jira/browse/HDDS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539049#comment-17539049 ]
Attila Doroszlai edited comment on HDDS-6763 at 5/18/22 8:00 PM:
-----------------------------------------------------------------
SCM during startup is waiting few seconds for {{addSCM}} call to complete. It is replacing DB with checkpoint in another thread. When SCM start continues and tries to access certificates in old, stale DB. Seems to be very similar to HDDS-6732. Also, similar problem was fixed in OM in HDDS-6685.
was (Author: adoroszlai):
SCM during startup is waiting few seconds for {{addSCM}} call to complete. It is replacing DB with checkpoint in another thread. When SCM start continues and tries to access certificates in old, stale DB. Seems to be very similar to HDDS-6732.
> SCM crashed while getting certificate from SCMCertStore
> -------------------------------------------------------
>
> Key: HDDS-6763
> URL: https://issues.apache.org/jira/browse/HDDS-6763
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Neil Joshi
> Priority: Major
> Attachments: docker-ozonesecure-ha.log, hs_err_pid8.log
>
>
> With secure HA cluster, scm crashes in safe-mode while retrieving certificate from SCMCertStore.
>
>
> {code:java}
> Current thread (0x00007fda6c017800): JavaThread "Listener at 0.0.0.0/9860" [_thread_in_native, id=171, stack(0x00007fda7399b000,0x00007fda73a9c000)]
> Stack: [0x00007fda7399b000,0x00007fda73a9c000], sp=0x00007fda73a99508, free space=1017k
> Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
> C 0x00007fda6d2426a0
> C [librocksdbjni6738631227401849863.so+0x28d8a2] Java_org_rocksdb_RocksDB_get__J_3BIIJ+0x62
> j org.rocksdb.RocksDB.get(J[BIIJ)[B+0
> j org.rocksdb.RocksDB.get(Lorg/rocksdb/ColumnFamilyHandle;[B)[B+13
> j org.apache.hadoop.hdds.utils.db.RDBTable.get([B)[B+9
> j org.apache.hadoop.hdds.utils.db.RDBTable.get(Ljava/lang/Object;)Ljava/lang/Object;+5
> j org.apache.hadoop.hdds.utils.db.TypedTable.getFromTable(Ljava/lang/Object;)Ljava/lang/Object;+14
> j org.apache.hadoop.hdds.utils.db.TypedTable.get(Ljava/lang/Object;)Ljava/lang/Object;+61
> j org.apache.hadoop.hdds.scm.server.SCMCertStore.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Ce
> rtificate;+17
> v ~StubRoutines::call_stub
> V [libjvm.so+0x8ce625] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x395
> V [libjvm.so+0xced613] invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) [clone .isra.199]+0x603
> V [libjvm.so+0xcee367] Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x137
> V [libjvm.so+0x97d953] JVM_InvokeMethod+0x1a3
> j jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0 java.base@11.0.14.1
> j jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+100 java.base@11.0.14.1
> J 3798 c1 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (10 bytes) @ 0x00007fda5556b674 [0x00007fda5556b540+0x000000
> 0000000134]
> J 3797 c1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (65 bytes) @ 0x00007fda5556b10c [0x00007fda5556ada0+0x000000000000036c]
> j org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+35
> j org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+33
> j com.sun.proxy.$Proxy21.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Certificate;+20
> j org.apache.hadoop.hdds.scm.server.StorageContainerManager.persistSCMCertificates()V+76
> j org.apache.hadoop.hdds.scm.server.StorageContainerManager.start()V+155
> j org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(Lorg/apache/hadoop/hdds/conf/OzoneConfiguration;)V+6
> j org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm()V+8
> {code}
> [^hs_err_pid8.log]
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org