You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/07/21 09:48:26 UTC
[GitHub] [ozone] JacksonYao287 opened a new pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
JacksonYao287 opened a new pull request #2447:
URL: https://github.com/apache/ozone/pull/2447
## What changes were proposed in this pull request?
generate correct yamls for running SCM-HA in k8s
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5471
Please replace this section with the link to the Apache JIRA)
## How was this patch tested?
k8s test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-884052504
@adoroszlai @elek please take a look! then new yaml file is generated by fleksizible using the updated Fleksizible file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-884052504
@adoroszlai @elek please take a look! then new yaml file is generated by fleksizible with the updated Fleksizible file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r681454688
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
no, the first scm instance will be started with --init, and the other two will be started will --bootstrap,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r682542001
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
@ChenSammi the logic here is as follows.
we want to create 3 scm replicas in this yaml file. for each scm instance, two init containers will be executed first, and then the container will be started. it seem like we have 3 individual scm , and for each scm, we do `ozone scm --init`, `ozone scm --bootstrap` and `ozone scm` sequentially one by one.
so for the first(primordial) scm, `ozone scm --init` succeeds , and the next `ozone scm --bootstrap` will fail with the message (SCM bootstrap command can only be executed in non-Primordial SCM), then it is started by `ozone scm` as a primordial scm.
for the second and third scm, `ozone scm --init` will fail with the message(SCM init command can only be executed in Primordial SCM), and `ozone scm --bootstrap` will succeed, and then it is started by `ozone scm` as a non-primordial scm
i am not sure is it clear to you ? please let me know if the description is no clear, thanks
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
@ChenSammi the logic here is as follows.
we want to create 3 scm replicas in this yaml file. for each scm instance, two init containers will be executed first, and then the container will be started. it seem like we have 3 individual scm , and for each scm, we do `ozone scm --init`, `ozone scm --bootstrap` and `ozone scm` sequentially one by one.
so for the first(primordial) scm, `ozone scm --init` succeeds , and the next `ozone scm --bootstrap` will fail with the message (SCM bootstrap command can only be executed in non-Primordial SCM), then it is started by `ozone scm` as a primordial scm.
for the second and third scm, `ozone scm --init` will fail with the message(SCM init command can only be executed in Primordial SCM), and `ozone scm --bootstrap` will succeed, and then it is started by `ozone scm` as a non-primordial scm
by this way , we got this scm HA cluster!
i am not sure is it clear to you ? please let me know if the description is no clear, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r681454688
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
no, the first scm instance will be started with --init, and the other two will be started will --bootstrap, this CRD file are used in the k8s cluster of my test environment and works well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r680635005
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
Do all three scm instances neet to start with --bootstrap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r682542001
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
@ChenSammi the logic here is as follows.
we want to create 3 scm replicas in this yaml file. for each scm instance, two init containers will be executed first, and then the container will be started. it seem like we have 3 individual scm , and for each scm, we do `ozone scm --init`, `ozone scm --bootstrap` and `ozone scm` sequentially one by one.
so for the first(primordial) scm, `ozone scm --init` succeeds , and the next `ozone scm --bootstrap` will fail with the message (SCM bootstrap command can only be executed in non-Primordial SCM), then it is started by `ozone scm` as a primordial scm.
for the second and third scm, `ozone scm --init` will fail with the message(SCM init command can only be executed in Primordial SCM), and `ozone scm --bootstrap` will succeed, and then it is started by `ozone scm` as a non-primordial scm
i am not sure is it clear to you ? please let me know if the description is no clear, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-1002989420
thanks @adoroszlai for the review and verification.
> However, additional SCMs failed to start with the following log
i have used this yaml file to deploy a scm-ha-enabled k8s cluster and run many chaos tests on it, it always works well.
I'll look into why it fails here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-999983594
@adoroszlai could you take a look at this? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-1001670441
Thanks @JacksonYao287 for the patch. I tried adding `test.sh` to the affected `ozone-ha` example (copied from `ozone` example) to verify the change. However, additional SCMs failed to start with the following log.
```
2021-12-27 13:49:00 ERROR StorageContainerManager:320 - Please make sure you have run 'ozone scm --init' command to generate all the required metadata to /data/metadata/scm or make sure you have run 'ozone scm --bootstrap' cmd to add the SCM to existing SCM HA group.
2021-12-27 13:49:00 ERROR StorageContainerManagerStarter:73 - SCM start failed with exception
org.apache.hadoop.hdds.scm.exceptions.SCMException: SCM not initialized due to storage config failure.
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:321)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:491)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:503)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:168)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:142)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:71)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:47)
```
But bootstrap reported to be successful:
```
STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG: host = scm-2/10.42.0.44
STARTUP_MSG: args = [--bootstrap]
...
2021-12-27 13:45:53 INFO StorageContainerManager:987 - SCM BootStrap is successful for ClusterID CID-b449e5a7-66ff-44eb-8a6e-bbf202cbf05e, SCMID 5053d1c5-7b10-4093-b19f-dcf28410a21f
2021-12-27 13:45:53 INFO StorageContainerManager:989 - Primary SCM Node ID ba47cc25-6474-4e64-a32e-baa5126f85fd
```
Can you please check?
Here's the run from my fork:
https://github.com/adoroszlai/hadoop-ozone/runs/4642413726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#issuecomment-884052504
@adoroszlai @elek please take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r680635005
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
Do all three sccm instances neet to start with --bootstrap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2447: HDDS-5471. generate correct yamls for running SCM-HA in k8s
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2447:
URL: https://github.com/apache/ozone/pull/2447#discussion_r682542001
##########
File path: hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/scm-statefulset.yaml
##########
@@ -52,6 +52,15 @@ spec:
volumeMounts:
- name: data
mountPath: /data
+ - name: bootstrap
+ image: '@docker.image@'
+ args:
+ - ozone
+ - scm
+ - --bootstrap
Review comment:
@ChenSammi the logic here is as follows.
we want to create 3 scm replicas in this yaml file. for each scm instance, two init containers will be executed first, and then the container will be started. it seem like we have 3 individual scm , and for each scm, we do `ozone scm --init`, `ozone scm --bootstrap` and `ozone scm` sequentially one by one.
so for the first(primordial) scm, `ozone scm --init` succeeds , and the next `ozone scm --bootstrap` will fail with the message (SCM bootstrap command can only be executed in non-Primordial SCM), then it is started by `ozone scm` as a primordial scm.
for the second and third scm, `ozone scm --init` will fail with the message(SCM init command can only be executed in Primordial SCM), and `ozone scm --bootstrap` will succeed, and then it is started by `ozone scm` as a non-primordial scm
by this way , we got this scm HA cluster!
i am not sure is it clear to you ? please let me know if the description is no clear, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org