You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/04/20 09:14:29 UTC
[GitHub] [ozone] elek opened a new pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
elek opened a new pull request #2165:
URL: https://github.com/apache/ozone/pull/2165
## What changes were proposed in this pull request?
Today ozonesecure compose clusters (and ozonesecure-ha and ozonesecure-mr) use an adhoc keytab issuer. The issuer is download during the image creation and uses a third party go lang application to create the keytabs on-demand.
As discussed earlier, it would be faster to use a dedicated, pre-built container image which includes the pre-created keytabs instead of issuing them on-the fly (keytab generation is slow + container creation is slow)
For each of the tagged images we can export to current keytabs to hadoop-ozone/dist/src/main/compose/ which can be mounted to compose clusters.
It makes the overall acceptance test faster (instead of creating keytab, which is quite slow, we can start the cluster immediately). And we don't need to depend on an external utility app.
Pre-created keytabs are also more similar to production environment...
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5123
## How was this patch tested?
CI is passed on my fork:
https://github.com/elek/ozone/actions/runs/763590947
The time between starting test.sh script and first robot test:
master: 3:30 (01:43:08 --01:46:38)
this patch: 2:10 (12:59:29 13:02:39)
(note: there are some variances between different builds, and in general the mentioned patch build was a slower one. It can be even faster).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2165:
URL: https://github.com/apache/ozone/pull/2165#discussion_r620046767
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
##########
@@ -78,7 +80,7 @@ MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.map.memory.mb=2048
MAPRED-SITE.XML_mapreduce.reduce.memory.mb=2048
#MAPRED-SITE.XML_mapred.child.java.opts=-Xmx2048
-MAPRED-SITE.XML_mapreduce.application.classpath=/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/ozone/share/ozone/lib/hadoop-ozone-filesystem-hadoop3-@project.version@.jar
+MAPRED-SITE.XML_mapreduce.application.classpath=/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/ozone/share/ozone/lib/hadoop-ozone-filesystem-hadoop3-1.1.0-SNAPSHOT.jar
Review comment:
Seems to be unintended change (copy from target to source).
##########
File path: hadoop-ozone/dist/src/main/smoketest/commonlib.robot
##########
@@ -32,6 +32,6 @@ Kinit HTTP user
Kinit test user
[arguments] ${user} ${keytab}
- ${hostname} = Execute hostname
+ ${hostname} = Execute hostname | sed 's/.org//' | sed 's/[0-9]//'
Review comment:
I think ACLs like `user:testuser2/${SCM}@EXAMPLE.COM` need to be tweaked, too. With SCM HA they apply to `testuser2/scm1`, but this `sed` makes the test `kinit` with `testuser2/scm`.
This causes test failure:
```
Can follow link with read access | FAIL |
'PERMISSION_DENIED User testuser2/scm@EXAMPLE.COM doesn't have READ permission to access volume 84112-target null null' does not contain 'key-in-readable-bucket'
```
https://github.com/apache/ozone/runs/2400748407#step:6:1636
##########
File path: hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
##########
@@ -120,6 +120,12 @@ run cp -p -r "${ROOT}/hadoop-ozone/dist/src/main/smoketest" .
run cp -p -r "${ROOT}/hadoop-ozone/dist/target/k8s" kubernetes
run cp -p -r "${ROOT}/hadoop-ozone/dist/target/Dockerfile" .
+#Copy pre-generated keytabs
+run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure
+run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-mr
+run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-om-ha
+run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-ha
Review comment:
Can we put a single copy under `compose/` (or a subdirectory) and use it for all environments?
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
##########
@@ -108,13 +112,15 @@ services:
- ozone
volumes:
- ../..:/opt/ozone
+ - ./keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
- ../../libexec/transformation.py:/opt/transformation.py
ports:
- 8088:8088
env_file:
- ./docker-config
environment:
- KERBEROS_KEYTABS: rm HTTP hadoop
+ HADOOP_CLASSPATH: /opt/ozone/share/ozone/lib/hadoop-ozone-filesystem-hadoop3-1.1.0-SNAPSHOT.jar
Review comment:
Is this needed (same variable is defined in `docker-config`)? If so, shouldn't version number come from placeholder?
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-compose.yaml
##########
@@ -55,8 +58,6 @@ services:
env_file:
- docker-config
environment:
- WAITFOR: scm3.org:9865
Review comment:
Are these `WAITFOR` configs being removed intentionally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2165:
URL: https://github.com/apache/ozone/pull/2165#issuecomment-851313675
The build is green, merging it now. Thanks, @adoroszlai the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek merged pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
Posted by GitBox <gi...@apache.org>.
elek merged pull request #2165:
URL: https://github.com/apache/ozone/pull/2165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2165:
URL: https://github.com/apache/ozone/pull/2165#discussion_r634754365
##########
File path: hadoop-ozone/dist/src/main/smoketest/commonlib.robot
##########
@@ -32,6 +32,6 @@ Kinit HTTP user
Kinit test user
[arguments] ${user} ${keytab}
- ${hostname} = Execute hostname
+ ${hostname} = Execute hostname | sed 's/.org//' | sed 's/[0-9]//'
Review comment:
Thanks @elek for addressing my other comments. I think this one still needs to be done.
```diff
--- hadoop-ozone/dist/src/main/smoketest/basic/links.robot
+++ hadoop-ozone/dist/src/main/smoketest/basic/links.robot
@@ -42,11 +42,11 @@
Execute ozone sh bucket link ${source}/readable-bucket ${target}/readable-link
Execute ozone sh bucket link ${source}/readable-bucket ${target}/unreadable-link
Execute ozone sh bucket link ${source}/unreadable-bucket ${target}/link-to-unreadable-bucket
- Execute ozone sh volume addacl --acl user:testuser2/${SCM}@EXAMPLE.COM:r ${target}
- Execute ozone sh volume addacl --acl user:testuser2/${SCM}@EXAMPLE.COM:rl ${source}
- Execute ozone sh bucket addacl --acl user:testuser2/${SCM}@EXAMPLE.COM:rl ${source}/readable-bucket
- Execute ozone sh bucket addacl --acl user:testuser2/${SCM}@EXAMPLE.COM:r ${target}/readable-link
- Execute ozone sh bucket addacl --acl user:testuser2/${SCM}@EXAMPLE.COM:r ${target}/link-to-unreadable-bucket
+ Execute ozone sh volume addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}
+ Execute ozone sh volume addacl --acl user:testuser2/scm@EXAMPLE.COM:rl ${source}
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:rl ${source}/readable-bucket
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}/readable-link
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}/link-to-unreadable-bucket
Can follow link with read access
Execute kdestroy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on a change in pull request #2165: HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #2165:
URL: https://github.com/apache/ozone/pull/2165#discussion_r630036304
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
##########
@@ -108,13 +112,15 @@ services:
- ozone
volumes:
- ../..:/opt/ozone
+ - ./keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
- ../../libexec/transformation.py:/opt/transformation.py
ports:
- 8088:8088
env_file:
- ./docker-config
environment:
- KERBEROS_KEYTABS: rm HTTP hadoop
+ HADOOP_CLASSPATH: /opt/ozone/share/ozone/lib/hadoop-ozone-filesystem-hadoop3-1.1.0-SNAPSHOT.jar
Review comment:
Good point. I am removing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org