You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "SaketaChalamchala (via GitHub)" <gi...@apache.org> on 2023/07/15 02:44:54 UTC
[GitHub] [ozone] SaketaChalamchala opened a new pull request, #5071: HDDS-8186. Ozone Client to accept replication config from params or config.
SaketaChalamchala opened a new pull request, #5071:
URL: https://github.com/apache/ozone/pull/5071
## What changes were proposed in this pull request?
The [documentation](https://ozone.apache.org/docs/1.3.0/start/runningviadocker.html) has instructions to create a single DN node Ozone docker environment. But, key create commands fail because server-side default replication config is RATIS THREE.
We set client replication config to 1 in client configuration in the docker environments but, the client seems to accept custom replication configs only if passed as params on the CLI like `ozone sh key put /vol1/bucket1/key1 xxx -t RATIS -r ONE`.
Updated the client to pick up custom replication config from CLI command params or client configuration before defaulting to server-side configs.
Provided an option in docker compose to set replication type using env variable `OZONE_REPLICATION_TYPE` and updated the documentation to set replication type to RATIS, replication factor defaults in to 1 in docker compose file. This would ensure that users following the documentation would be able to spin up a single DN working Ozone cluster.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8186
## How was this patch tested?
Manual.
Follow [documentation](https://ozone.apache.org/docs/1.3.0/start/runningviadocker.html) to spin up a docker cluster.
Run
```
ozone sh volume create /vol1
ozone sh create bucket create /vol1/bucket1
touch /tmp/myfile
ozone sh key put /vol1/bucket1/key1 /tmp/myfile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5071: HDDS-8186. `ozone sh key put` does not work in single-datanode compose example
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5071:
URL: https://github.com/apache/ozone/pull/5071#issuecomment-1643357829
@umamaheswararao please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on a diff in pull request #5071: HDDS-8186. Ozone Client to accept replication config from params or config.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #5071:
URL: https://github.com/apache/ozone/pull/5071#discussion_r1265959322
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java:
##########
@@ -109,8 +110,12 @@ public void execute(OzoneClient client, OzoneAddress address)
}
}
- replication.fromParams(getConf()).ifPresent(config ->
- bb.setDefaultReplicationConfig(new DefaultReplicationConfig(config)));
+ ReplicationConfig replicationConfig =
+ replication.fromParamsOrConfig(getConf());
+ if (replicationConfig != null) {
+ bb.setDefaultReplicationConfig(
+ new DefaultReplicationConfig(replicationConfig));
+ }
Review Comment:
Yes, you're right. This should be a simple docker compose change. I updated the patch. Thanks @adoroszlai !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #5071: HDDS-8186. `ozone sh key put` does not work in single-datanode compose example
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #5071:
URL: https://github.com/apache/ozone/pull/5071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #5071: HDDS-8186. Ozone Client to accept replication config from params or config.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5071:
URL: https://github.com/apache/ozone/pull/5071#discussion_r1266893833
##########
hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml:
##########
@@ -28,6 +28,7 @@ x-common-config:
x-replication:
&replication
OZONE-SITE.XML_ozone.replication: ${OZONE_REPLICATION_FACTOR:-1}
+ OZONE-SITE.XML_ozone.replication.type: RATIS
Review Comment:
HDDS-9035 turned out to be invalid. On second look, I think we should set server-side default replication instead of client-side config here:
```suggestion
OZONE-SITE.XML_ozone.server.default.replication: ${OZONE_REPLICATION_FACTOR:-1}
```
This new property was introduced for HDDS-5293. I think that PR should have updated the docker compose definitions, too.
We can make the same change in all `docker-config` and `docker-compose.yaml` files, see https://github.com/adoroszlai/hadoop-ozone/commit/12cda5bd862d7b0eb38f2f08277cd029288b3ba5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on a diff in pull request #5071: HDDS-8186. `ozone sh key put` does not work in single-datanode compose example
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #5071:
URL: https://github.com/apache/ozone/pull/5071#discussion_r1267264845
##########
hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml:
##########
@@ -28,6 +28,7 @@ x-common-config:
x-replication:
&replication
OZONE-SITE.XML_ozone.replication: ${OZONE_REPLICATION_FACTOR:-1}
+ OZONE-SITE.XML_ozone.replication.type: RATIS
Review Comment:
Thanks @adoroszlai I can close this PR if you want to create a new PR from https://github.com/adoroszlai/hadoop-ozone/commit/12cda5bd862d7b0eb38f2f08277cd029288b3ba5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #5071: HDDS-8186. `ozone sh key put` does not work in single-datanode compose example
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #5071:
URL: https://github.com/apache/ozone/pull/5071#issuecomment-1661104612
> An improvement here could be to read value of --scale and set the config. I'm not sure if this is possible but just a suggestion.
I don't think this can be done @sadanand48 . To use RATIS THREE you would have to do
`OZONE_REPLICATION_FACTOR=3 docker-compose up --scale datanode=3 -d`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5071: HDDS-8186. Ozone Client to accept replication config from params or config.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5071:
URL: https://github.com/apache/ozone/pull/5071#issuecomment-1640134013
Thanks @SaketaChalamchala for updating the patch. I think it does what is intended. However, it uncovered a bug in Ozone CLI related to EC (HDDS-9035), which I'll need to fix before this can be merged. I'll mark this as draft until then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #5071: HDDS-8186. Ozone Client to accept replication config from params or config.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5071:
URL: https://github.com/apache/ozone/pull/5071#discussion_r1264350571
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java:
##########
@@ -109,8 +110,12 @@ public void execute(OzoneClient client, OzoneAddress address)
}
}
- replication.fromParams(getConf()).ifPresent(config ->
- bb.setDefaultReplicationConfig(new DefaultReplicationConfig(config)));
+ ReplicationConfig replicationConfig =
+ replication.fromParamsOrConfig(getConf());
+ if (replicationConfig != null) {
+ bb.setDefaultReplicationConfig(
+ new DefaultReplicationConfig(replicationConfig));
+ }
Review Comment:
Bucket-level replication config should be set only if explicitly specified via params. Otherwise it is left unset, and should fall back at server-side to the cluster-level default replication setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org