You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/11/10 19:07:40 UTC
[GitHub] [ozone] xBis7 opened a new pull request, #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
xBis7 opened a new pull request, #3948:
URL: https://github.com/apache/ozone/pull/3948
## What changes were proposed in this pull request?
In a secure cluster, if `ozone.metadata.dirs` are not specified then SCM will fallback to `hdds.datanode.dir` and will end up storing its certificates under the datanode path. It's preferable to let the system crash, rather than storing scm data under the datanode disk. Also, the datanodes crash if `ozone.metadata.dirs` aren't specified, so it makes sense to not have a fallback.
from `master` under `compose/ozonesecure`, edit `docker-config`
```
# OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.om.ratis.storage.dir=/data/om/ratis
OZONE-SITE.XML_ozone.om.db.dirs=/data/om/db
OZONE-SITE.XML_ozone.om.ratis.snapshot.dir=/data/om/ratis-snap
OZONE-SITE.XML_ozone.scm.ha.ratis.storage.dir=/data/scm/ratis
OZONE-SITE.XML_ozone.scm.db.dirs=/data/scm/db
OZONE-SITE.XML_ozone.recon.db.dir=/data/recon/db
OZONE-SITE.XML_ozone.recon.om.db.dir=/data/recon/om-db
OZONE-SITE.XML_ozone.recon.scm.db.dirs=/data/recon/scm-db
OZONE-SITE.XML_dfs.container.ratis.datanode.storage.dir=/data/dn/ratis
...
...
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
```
connect to scm and check files under `/data/hdds`
```
> docker-compose up --scale datanode=3 -d
> docker exec -it ozonesecure_scm_1 bash
bash-4.2$ ls -lah /data/hdds/scm/sub-ca/certs
total 20K
drwxr-xr-x 2 hadoop hadoop 4.0K Nov 10 19:03 .
drwxr-xr-x 4 hadoop hadoop 4.0K Nov 10 19:03 ..
-rwx------ 1 hadoop hadoop 1.3K Nov 10 19:03 36633700951438.crt
-rwx------ 1 hadoop hadoop 1.3K Nov 10 19:03 CA-1.crt
-rwx------ 1 hadoop hadoop 1.3K Nov 10 19:03 certificate.crt
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7476
## How was this patch tested?
This patch was tested manually in a docker cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
neils-dev commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1311041646
Thanks @xBis7 for raising this issue and submitting the PR.
> Also, the datanodes crash if `ozone.metadata.dirs `aren't specified,
Can you explain? The datanodes crash if this is not set? The `ozone.metadata.dirs` are optional fallback and recommended to not be set in production environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1314620934
Here is my understanding, not sure if this is totally right:
On the datanode, this change is safe and desirable. The datanode will fail to start if ozone.metadata.dirs is not specified, so hdds.datanode.dir never gets used. We would never want cert info on hdds.datanode.dir anyways since this is specified multiple times for each datanode volume. The instance that gets used is somewhat non-deterministic and if that volume fails the cert info will be gone.
On other components, I am not sure. If the cert directory they try to use is empty, will they get new ones from SCM?
It seems like there should be a way for components to specify their preferred storage directory. Then for compatibility inside the SecurityConfig, we can use existing defaults if that is not set. However, this would cause the cert directory to change on upgrade if the priority of existing configs was changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3948:
URL: https://github.com/apache/ozone/pull/3948#discussion_r1020289018
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java:
##########
@@ -123,10 +122,8 @@ public SecurityConfig(ConfigurationSource configuration) {
// Please Note: To make it easy for our customers we will attempt to read
// HDDS metadata dir and if that is not set, we will use Ozone directory.
- // TODO: We might want to fix this later.
- this.metadatDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
- configuration.get(OZONE_METADATA_DIRS,
- configuration.get(HDDS_DATANODE_DIR_KEY)));
+ this.metadataDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
+ configuration.get(OZONE_METADATA_DIRS));
Review Comment:
We could make the log more specific so that it includes the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on a diff in pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
neils-dev commented on code in PR #3948:
URL: https://github.com/apache/ozone/pull/3948#discussion_r1019696973
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java:
##########
@@ -123,10 +122,8 @@ public SecurityConfig(ConfigurationSource configuration) {
// Please Note: To make it easy for our customers we will attempt to read
// HDDS metadata dir and if that is not set, we will use Ozone directory.
- // TODO: We might want to fix this later.
- this.metadatDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
- configuration.get(OZONE_METADATA_DIRS,
- configuration.get(HDDS_DATANODE_DIR_KEY)));
+ this.metadataDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
+ configuration.get(OZONE_METADATA_DIRS));
Review Comment:
We can add a null check here, to log the error where it originates, for the `metadataDir `should both the `hdds.metadata.dir` ozone-site property and the fallback `ozone.metadata.dirs `is not set, such as
https://github.com/apache/ozone/blob/fe748d6776fc33124dad2d6cfc5f5dceee038a7f/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/CRLDBDefinition.java#L77
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 merged pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
errose28 merged PR #3948:
URL: https://github.com/apache/ozone/pull/3948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1322391937
@errose28 if this PR is ready to merge, can you merge this in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1315384447
@fapifta @errose28 Thanks for taking a look.
> DefaultCertificateClient also uses it to determine where to store the certificate of an entity, and where to store the key material for the certificate.
The implementation of these certificate client methods will have an effect for SCM, Ozone Manager, Ozone Recon, and Ozone DataNodes, as none of the component specific certificate client overrides these methods from the DefaultCertificateClient.
@fapifta I don't think that this change affects any of this.
That's the current code
```
this.metadatDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
configuration.get(OZONE_METADATA_DIRS,
configuration.get(HDDS_DATANODE_DIR_KEY)));
```
`HDDS_METADATA_DIR_NAME` is the default config and if it's not set then `OZONE_METADATA_DIRS` is used. If `OZONE_METADATA_DIRS` is not set then `HDDS_DATANODE_DIR_KEY` will be used. We are only removing this last option.
In any case, if `OZONE_METADATA_DIRS` is not set, then all the datanodes will crash and we will end up having a cluster where all datanodes are down and SCM will be using a datanode disk to store it's certificates.
If we have multiple volumes as @errose28 mentioned
> We would never want cert info on hdds.datanode.dir anyways since this is specified multiple times for each datanode volume
The config gets the path for the last volume and then stores the certificates under it.
> It seems like there should be a way for components to specify their preferred storage directory.
From `HddsConfigKeys`
```
// In any real cluster where performance matters, the SCM, OM and DN
// metadata locations must be configured explicitly.
public static final String OZONE_METADATA_DIRS = "ozone.metadata.dirs";
```
But that doesn't seem to be the case, since we can't really avoid specifying `ozone.metadata.dirs`. I agree that every component should have it's own metadata dir. Currently, we are setting a db dir and a ratis dir.
> Also it is an interesting philosophical question whether HDDS_METADATA_DIR or OZONE_METADATA_DIR should take precedence for OM and for Recon, as those are Ozone services, while which should take precedence for SCM and DN as those are HDDS services... :)
Since we already have other directories for each component, what if there was also a metadata dir? For instance, for scm we have
```
ozone.scm.ha.ratis.storage.dir
ozone.scm.db.dirs
```
and there would also be an `ozone.scm.metadata.dir` config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1314099534
cc @aswinshakil @swamirishi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
neils-dev commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1316183275
> Also, the datanodes crash if ozone.metadata.dirs aren't specified,
The ozone.metadata.dirs are optional fallback and recommended to not be set in production environments.
> It doesn't seem optional for the datanode. It might not be used but it has to be specified. If not specified, then we will get a null pointer exception in HddsDatanodeService.start() while initializing [DatanodeLayoutStorage](https://github.com/xBis7/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java#L267).
This looks like a potential bug here, I would expect the datanodes to not need the ozone.metadata.dirs to be set, only to use it as a fallback. @errose28 is there a reason for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1320639167
> I would expect the datanodes to not need the ozone.metadata.dirs to be set, only to use it as a fallback
Yes @neils-dev this existing part of the code is definitely not desirable. We should try to get to a point where `ozone.metadata.dirs` is truly optional, but that is tangental to the change in this PR, since this config is not being removed here. Let's continue a discussion of how to reach that point in a separate Jira. Probably related to HDDS-2025.
For this change the `CRLDBDefinition` LGTM since it only used on the datanode, but I have a question on the `SecurityConfig`. Say there's a cluster with `ozone.metadata.dir` defined in the datanode config only, and neither `ozone.metadata.dir` or `hdds.metadata.dir` defined on the SCM, so its certificate info is written to `hdds.datanode.dir`.
1. What will happen to the cluster when it is restarted with this change? The user would be prompted to set one of the existing configs, but what if they set it to a different directory than SCM was previously using, so SCM has no cert information? @fapifta do you know how the primordial SCM will handle this?
2. Even if the above causes a problem, is it worth handling this somehow? It seems like a corner case and not worth leaving the `hdds.datanode.dir` config check in just for that. I just want to make sure it is recoverable somehow if it does happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1321813633
> What will happen to the cluster when it is restarted with this change? The user would be prompted to set one of the existing configs, but what if they set it to a different directory than SCM was previously using, so SCM has no cert information?
@errose28 with the current code, let's say that we have a cluster running and SCM uses `ozone_metadata_dirs` to store cert info. If we change the location for `ozone_metadata_dirs` config then SCM will use that location instead. If we remove the config and restart the SCM, then it will crash. If we replace the config with `hdds.metadata.dir`, then it will use that location instead. Everytime the SCM shuts down, it removes all certs and then recreates them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1311751986
> The datanodes crash if this is not set? The ozone.metadata.dirs are optional fallback and recommended to not be set in production environments.
Hey @neils-dev thanks for looking into this. It doesn't seem optional for the datanode. It might not be used but it has to be specified. If not specified, then we will get a null pointer exception in `HddsDatanodeService.start()` while initializing [DatanodeLayoutStorage](https://github.com/xBis7/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java#L267). The `DatanodeLayoutStorage` [constructor](https://github.com/xBis7/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/DatanodeLayoutStorage.java#L46) passes to the parent class the parameter `ServerUtils.getOzoneMetaDirPath(conf)` which in return calls this [method](https://github.com/xBis7/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java#L194-L207) that relies only on the config value for `ozone.metadata.dirs`. There is no fallback or an alternative value to be used, so if
it's null then the datanode service won't start and the datanode shuts down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3948:
URL: https://github.com/apache/ozone/pull/3948#discussion_r1020288228
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java:
##########
@@ -123,10 +122,8 @@ public SecurityConfig(ConfigurationSource configuration) {
// Please Note: To make it easy for our customers we will attempt to read
// HDDS metadata dir and if that is not set, we will use Ozone directory.
- // TODO: We might want to fix this later.
- this.metadatDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
- configuration.get(OZONE_METADATA_DIRS,
- configuration.get(HDDS_DATANODE_DIR_KEY)));
+ this.metadataDir = this.configuration.get(HDDS_METADATA_DIR_NAME,
+ configuration.get(OZONE_METADATA_DIRS));
Review Comment:
There are already null checks for this in `SecurityConfig`. Check [getKeyLocation()](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java#L273-L277) and [getCertificateLocation()](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java#L286-L290).
With the new change and without `ozone.metadata.dirs` specified, that's the output in the logs
```
2022-11-11 14:45:16,642 [main] INFO ha.HASecurityUtils: Initializing secure StorageContainerManager.
Metadata directory can't be null. Please check configs.
2022-11-11 14:45:16,992 [shutdown-hook-0] INFO server.StorageContainerManagerStarter: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down StorageContainerManager at scm/172.22.0.10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] fapifta commented on pull request #3948: HDDS-7476. SCM root CA stored under hdds.datanode.dir if ozone.metadata.dirs is not specified
Posted by GitBox <gi...@apache.org>.
fapifta commented on PR #3948:
URL: https://github.com/apache/ozone/pull/3948#issuecomment-1314521457
Hi @xBis7,
I don't think we can merge this change implemented this way...
If you look at the usages of getCertificateLocation, and getKeyLocation in SecurityConfig, then you will find that beside the CA server code, the DefaultCertificateClient also uses it to determine where to store the certificate of an entity, and where to store the key material for the certificate.
The implementation of these certificate client methods will have an effect for SCM, Ozone Manager, Ozone Recon, and Ozone DataNodes, as none of the component specific certificate client overrides these methods from the DefaultCertificateClient.
The approach in the current code is bad, as it seems to rely on what config option is set in the configuration of which service, and it may cause problems, but this fix would make DataNodes to fail to find their correct location even if it is the only thing set in their config, at least as I understand the code looking at it quickly. Did I misinterpret the code and the change?
Also it is an interesting philosophical question whether HDDS_METADATA_DIR or OZONE_METADATA_DIR should take precedence for OM and for Recon, as those are Ozone services, while which should take precedence for SCM and DN as those are HDDS services... :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org