You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2023/03/21 17:49:19 UTC
[GitHub] [ozone] adoroszlai opened a new pull request, #4444: HDDS-8219. Run HA secure tests from s3g container
adoroszlai opened a new pull request, #4444:
URL: https://github.com/apache/ozone/pull/4444
## What changes were proposed in this pull request?
In `ozonesecure-ha` environment SCM containers have hostnames like `scm1.org`, but all Kerberos principals for SCM nodes are in the form `testuser/scm@EXAMPLE.COM`. Since no host named `scm` exists, lookup takes quite some time:
```
$ OZONE_LOGLEVEL=TRACE ozone sh volume create /vol1 |& grep 'lookup'
[main] TRACE security.SecurityUtil: Name lookup for om2 took 1 ms.
[main] TRACE security.SecurityUtil: Name lookup for om3 took 0 ms.
[main] TRACE security.SecurityUtil: Name lookup for om1 took 0 ms.
[main] TRACE security.SecurityUtil: Name lookup for scm took 497 ms.
```
We can improve this simply by executing the same commands from another container, e.g. `s3g`:
```
$ OZONE_LOGLEVEL=TRACE ozone sh volume create /vol2 |& grep 'lookup'
[main] TRACE security.SecurityUtil: Name lookup for om2 took 1 ms.
[main] TRACE security.SecurityUtil: Name lookup for om3 took 0 ms.
[main] TRACE security.SecurityUtil: Name lookup for om1 took 0 ms.
[main] TRACE security.SecurityUtil: Name lookup for s3g took 0 ms.
```
Tests for bucket links need a minor change to remove hard-coded `scm` from permissions in test setup.
Opening as draft, since this is based on HDDS-7975 (#4437).
https://issues.apache.org/jira/browse/HDDS-8219
## How was this patch tested?
Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4482200047/jobs/7879923515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4444:
URL: https://github.com/apache/ozone/pull/4444#discussion_r1144019169
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -47,6 +48,7 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
+OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true
Review Comment:
Yes, Ozone FS tests have a test case for write with streaming, which would fail without this.
https://github.com/apache/ozone/blob/030cf40fb1252f65de238e50ee13e6c0be58ef7e/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot#L73-L77
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #4444:
URL: https://github.com/apache/ozone/pull/4444#discussion_r1144016199
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -47,6 +48,7 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
+OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true
Review Comment:
It this needed for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4444:
URL: https://github.com/apache/ozone/pull/4444#discussion_r1144339120
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -47,6 +48,7 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
+OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true
Review Comment:
These are two separate settings. `dfs.container.ratis.datastream.enabled=true` enables it at the server-side, but it's up to the client to decide if it uses regular or streaming path. For FS API it is achieved by setting `ozone.fs.datastream.enabled`. ([docs](https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/feature/streaming-write-pipeline.html))
In short, if we want to run `ozonefs.robot` in some compose environment, we need to add this config. It's just copied from the `ozone` and `ozonesecure` environments to the HA ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #4444:
URL: https://github.com/apache/ozone/pull/4444#discussion_r1144193417
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -47,6 +48,7 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
+OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true
Review Comment:
That looks like a specific invocation to enable streaming vs. Setting it for all ozone ha tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4444:
URL: https://github.com/apache/ozone/pull/4444#discussion_r1144426331
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -47,6 +48,7 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
+OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true
Review Comment:
BTW, you are right, the config change is not needed for _this_ PR, rather the one this is based on (see description). Now that the base PR is merged, I've updated this one, too, and now the change is reduced. Sorry for the confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4444:
URL: https://github.com/apache/ozone/pull/4444#issuecomment-1483774655
Thanks @kerneltime, @smengcl for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #4444: HDDS-8219. Run HA secure tests from s3g container
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4444:
URL: https://github.com/apache/ozone/pull/4444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org