You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ritesh H Shukla (Jira)" <ji...@apache.org> on 2022/05/17 22:38:00 UTC
[jira] [Commented] (HDDS-6763) SCM crashed while getting certificate from SCMCertStore
[ https://issues.apache.org/jira/browse/HDDS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538500#comment-17538500 ]
Ritesh H Shukla commented on HDDS-6763:
---------------------------------------
Which version of RocksDB is this? Can you also capture the SCM logs? There are 2 types of crashes we have seen, one around memory access faults which are due to bugs in RocksDB, and the other due to invalid handles being used in the Java code. This most likely is due to an invalid handle.
> SCM crashed while getting certificate from SCMCertStore
> -------------------------------------------------------
>
> Key: HDDS-6763
> URL: https://issues.apache.org/jira/browse/HDDS-6763
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Neil Joshi
> Priority: Major
> Attachments: hs_err_pid8.log
>
>
> With secure HA cluster, scm crashes in safe-mode while retrieving certificate from SCMCertStore.
>
>
> {code:java}
> Current thread (0x00007fda6c017800): JavaThread "Listener at 0.0.0.0/9860" [_thread_in_native, id=171, stack(0x00007fda7399b000,0x00007fda73a9c000)]
> Stack: [0x00007fda7399b000,0x00007fda73a9c000], sp=0x00007fda73a99508, free space=1017k
> Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
> C 0x00007fda6d2426a0
> C [librocksdbjni6738631227401849863.so+0x28d8a2] Java_org_rocksdb_RocksDB_get__J_3BIIJ+0x62
> j org.rocksdb.RocksDB.get(J[BIIJ)[B+0
> j org.rocksdb.RocksDB.get(Lorg/rocksdb/ColumnFamilyHandle;[B)[B+13
> j org.apache.hadoop.hdds.utils.db.RDBTable.get([B)[B+9
> j org.apache.hadoop.hdds.utils.db.RDBTable.get(Ljava/lang/Object;)Ljava/lang/Object;+5
> j org.apache.hadoop.hdds.utils.db.TypedTable.getFromTable(Ljava/lang/Object;)Ljava/lang/Object;+14
> j org.apache.hadoop.hdds.utils.db.TypedTable.get(Ljava/lang/Object;)Ljava/lang/Object;+61
> j org.apache.hadoop.hdds.scm.server.SCMCertStore.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Ce
> rtificate;+17
> v ~StubRoutines::call_stub
> V [libjvm.so+0x8ce625] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x395
> V [libjvm.so+0xced613] invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) [clone .isra.199]+0x603
> V [libjvm.so+0xcee367] Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x137
> V [libjvm.so+0x97d953] JVM_InvokeMethod+0x1a3
> j jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0 java.base@11.0.14.1
> j jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+100 java.base@11.0.14.1
> J 3798 c1 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (10 bytes) @ 0x00007fda5556b674 [0x00007fda5556b540+0x000000
> 0000000134]
> J 3797 c1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@11.0.14.1 (65 bytes) @ 0x00007fda5556b10c [0x00007fda5556ada0+0x000000000000036c]
> j org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+35
> j org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+33
> j com.sun.proxy.$Proxy21.getCertificateByID(Ljava/math/BigInteger;Lorg/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore$CertType;)Ljava/security/cert/X509Certificate;+20
> j org.apache.hadoop.hdds.scm.server.StorageContainerManager.persistSCMCertificates()V+76
> j org.apache.hadoop.hdds.scm.server.StorageContainerManager.start()V+155
> j org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(Lorg/apache/hadoop/hdds/conf/OzoneConfiguration;)V+6
> j org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm()V+8
> {code}
> [^hs_err_pid8.log]
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org