You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/10/14 05:26:08 UTC
[GitHub] [ozone] neils-dev opened a new pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
neils-dev opened a new pull request #2739:
URL: https://github.com/apache/ozone/pull/2739
… error responses to client on error (fixing general 500 response in current gateway).
## What changes were proposed in this pull request?
S3 gateway Gprc support for all ozone s3 supported client requests with security and ACLs enabled. This PR adds s3g Grpc acl support with patches to the `OMClientRequest` user thread context and to the `ozone manager`.
Included in this PR are enhancements to the s3 gateway client to return meaningful error responses, http return codes and descriptions, to the caller on error. For example a user accessing a resource **_without_** permission is returned `HTTP 403` `AccessDenied`. Previously all errors on client creation return `HTTP 500`.
All secure cluster smoke tests are re-enabled in this PR. Secure cluster is updated to enable s3g Grpc and to enable all s3 gateway acceptance tests.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5781
## How was this patch tested?
Manual Testing with secure ozone cluster:
s3 request - ozone cluster processing s3 create bucket request with permission _and without_ permission.
With secret key:
`$ cd hadoop-ozone/dist/target/ozone-1.2.0-SNAPSHOT/compose/ozonesecure`
`$ docker-compose up -d --scale datanode=3`
`$ docker-compose run scm bash`
`bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm`
`bash-4.2$ ozone s3 getsecret`
`$ export AWS_ACCESS_KEY=testuser/scm@EXAMPLE.COM` `AWS_SECRET_KEY=<key>`
`$ aws s3api --endpoint http://localhost:9878 create-bucket --bucket=bucket1`
_{
"Location": "http://localhost:9878/bucket1"
}_
Without secret key:
`$ export AWS_ACCESS_KEY=`
`$ export AWS_SECRET_KEY=`
`$ aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1`
_An error occurred (AccessDenied) when calling the CreateBucket operation: User doesn't have the right to access this resource._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#discussion_r738696111
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -273,15 +278,16 @@ public UserGroupInformation createUGI() {
if (userGroupInformation != null) {
return userGroupInformation;
}
-
if (omRequest.hasUserInfo() &&
!StringUtils.isBlank(omRequest.getUserInfo().getUserName())) {
userGroupInformation = UserGroupInformation.createRemoteUser(
omRequest.getUserInfo().getUserName());
+ LOG.debug("creating UGI remote user for acl");
Review comment:
Minor NIT: No much useful from log, can be skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#discussion_r740393312
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -4038,7 +4038,7 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
if (isAclEnabled) {
InetAddress remoteIp = Server.getRemoteIp();
resolved = resolveBucketLink(requested, new HashSet<>(),
- Server.getRemoteUser(),
+ getRemoteUser(),
Review comment:
@neils-dev ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#issuecomment-958089252
Thank You @neils-dev for the offline discussion and explaination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on a change in pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
neils-dev commented on a change in pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#discussion_r739888167
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -273,15 +278,16 @@ public UserGroupInformation createUGI() {
if (userGroupInformation != null) {
return userGroupInformation;
}
-
if (omRequest.hasUserInfo() &&
!StringUtils.isBlank(omRequest.getUserInfo().getUserName())) {
userGroupInformation = UserGroupInformation.createRemoteUser(
omRequest.getUserInfo().getUserName());
+ LOG.debug("creating UGI remote user for acl");
Review comment:
Updated with latest commit to omit log statement. Thanks.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -273,15 +278,16 @@ public UserGroupInformation createUGI() {
if (userGroupInformation != null) {
return userGroupInformation;
}
-
if (omRequest.hasUserInfo() &&
!StringUtils.isBlank(omRequest.getUserInfo().getUserName())) {
userGroupInformation = UserGroupInformation.createRemoteUser(
omRequest.getUserInfo().getUserName());
+ LOG.debug("creating UGI remote user for acl");
return userGroupInformation;
} else {
// This will never happen, as for every OM request preExecute, we
// should add userInfo.
+ LOG.debug("NO UGI for acl");
Review comment:
Also updated with latest commit to omit log statement. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 merged pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #2739:
URL: https://github.com/apache/ozone/pull/2739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 merged pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #2739:
URL: https://github.com/apache/ozone/pull/2739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#issuecomment-958089252
Thank You @neils-dev for the offline discussion and explaination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#discussion_r738701377
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -4038,7 +4038,7 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
if (isAclEnabled) {
InetAddress remoteIp = Server.getRemoteIp();
resolved = resolveBucketLink(requested, new HashSet<>(),
- Server.getRemoteUser(),
+ getRemoteUser(),
Review comment:
Not related to this PR, but i have a question which i want to understand.
In GrpcContext how the UserGroupInformation.getCurrentUser() is being populated here?
I see below code in OzoneManagerServiceGrpc, but not understood how getCurrentUser will be populated with S3 accesskey id?
```
try {
omResponse =
UserGroupInformation.getCurrentUser().doAs(
(PrivilegedExceptionAction<OMResponse>) () -> {
try {
return this.omTranslator.
submitRequest(NULL_RPC_CONTROLLER, request);
} catch (Throwable se) {
Throwable e = se.getCause();
if (e == null) {
throw new IOException(se);
} else {
throw e instanceof IOException ?
(IOException) e : new IOException(se);
}
}
});
}
```
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -273,15 +278,16 @@ public UserGroupInformation createUGI() {
if (userGroupInformation != null) {
return userGroupInformation;
}
-
if (omRequest.hasUserInfo() &&
!StringUtils.isBlank(omRequest.getUserInfo().getUserName())) {
userGroupInformation = UserGroupInformation.createRemoteUser(
omRequest.getUserInfo().getUserName());
+ LOG.debug("creating UGI remote user for acl");
return userGroupInformation;
} else {
// This will never happen, as for every OM request preExecute, we
// should add userInfo.
+ LOG.debug("NO UGI for acl");
Review comment:
Minor NIT: No much useful from log. Same as above
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##########
@@ -273,15 +278,16 @@ public UserGroupInformation createUGI() {
if (userGroupInformation != null) {
return userGroupInformation;
}
-
if (omRequest.hasUserInfo() &&
!StringUtils.isBlank(omRequest.getUserInfo().getUserName())) {
userGroupInformation = UserGroupInformation.createRemoteUser(
omRequest.getUserInfo().getUserName());
+ LOG.debug("creating UGI remote user for acl");
Review comment:
Minor NIT: No much useful from log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2739: HDDS 5781. Enable ACLs and support for all s3 file operations.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2739:
URL: https://github.com/apache/ozone/pull/2739#discussion_r740393312
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -4038,7 +4038,7 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
if (isAclEnabled) {
InetAddress remoteIp = Server.getRemoteIp();
resolved = resolveBucketLink(requested, new HashSet<>(),
- Server.getRemoteUser(),
+ getRemoteUser(),
Review comment:
@neils-dev ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org