You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Neil Joshi (Jira)" <ji...@apache.org> on 2023/02/17 01:30:00 UTC
[jira] [Created] (HDDS-7985) [SCM HA] On SCM Disk failure recovery causes Datanode Failure on startup
Neil Joshi created HDDS-7985:
--------------------------------
Summary: [SCM HA] On SCM Disk failure recovery causes Datanode Failure on startup
Key: HDDS-7985
URL: https://issues.apache.org/jira/browse/HDDS-7985
Project: Apache Ozone
Issue Type: Bug
Reporter: Neil Joshi
Recovery from an SCM disk failure when no backup is avail requires,
* Clean _ozone.scm.db.dirs_ __ and __ _ozone.metadata.dirs_ locations
and bootstrapping the SCM. Whether SCM is primodial or not an error occurs when recovering from a failed disk with no backup when starting a datanode after SCM recovery.
Datanodes brought up after SCM disk failure recovery are unable to start due to a CA certificate error observed, stating the number of certificates received from the SCM is greater than the number expected:
{code:java}
ozonesecure-ha-datanode1-1 | 2023-02-17 00:46:40 INFO HAUtils:457 - Expected CA list size 4, where as received CA List size 5.{code}
In this case when listing the certificates stored by the SCM, it reports a total of 5 scm certificates after SCM2 recovers from disk failure:
{code:java}
CN=scm@scm1.org
CN=scm-sub@scm1.org
CN=scm-sub@scm2.org
CN=scm-sub@scm3.org
CN=scm-sub@scm2.org
{code}
It appears to have 2 entries for SCM 2 (the scm disk failure recovery node)
$ ozone admin certs list
bash-4.2$ ozone admin cert list
{code:java}
Total 12 valid certificates:
SerialNumber Valid From Expiry Subject
1 Fri Feb 17 00:00:00 UTC 2023 Mon Mar 27 00:00:00 UTC 2028 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=scm@scm1.org
10760186198072 Fri Feb 17 00:00:00 UTC 2023 Mon Mar 27 00:00:00 UTC 2028 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=scm-sub@scm1.org
10779888473070 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=recon@recon
10780166036417 Fri Feb 17 00:00:00 UTC 2023 Mon Mar 27 00:00:00 UTC 2028 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f99f1a81-7cce-44c9-a09b-9f7bbc48b6ac, CN=scm-sub@scm2.org
10788394717480 Fri Feb 17 00:00:00 UTC 2023 Mon Mar 27 00:00:00 UTC 2028 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=598be6bc-7d86-4cab-84dc-668a162a7ec2, CN=scm-sub@scm3.org
10800769855768 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=dn@bd3138308a3f
10801305457014 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=dn@e4795cc77124
10801871334038 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=dn@3eb28ff965a1
10803980992569 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=om2
10804543987939 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=om3
10806118720884 Fri Feb 17 00:00:00 UTC 2023 Sat Feb 17 00:00:00 UTC 2024 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=f02b032a-7da0-4132-8a31-61c3d078e6cb, CN=om1
10932809284268 Fri Feb 17 00:00:00 UTC 2023 Mon Mar 27 00:00:00 UTC 2028 O=CID-abb46225-77ba-4132-ac6e-96792b40450c, OU=b4a175f3-c6a4-47fd-bcc5-c081b03de8c7, CN=scm-sub@scm2.org {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org