You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/06/23 07:38:22 UTC

[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

maobaolong opened a new pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110


   ## What changes were proposed in this pull request?
   
   NPE shouldn't appear in the log.
   
   ## What is the link to the Apache JIRA
   HDDS-3843
   
   ## How was this patch tested?
   
   Start a s3g without enable the security.
   
   Start a goofys without Authorization related setting.
   
   ls the dir within the mount point, the NPE will appear. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] closed pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-696453432


   Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
   
   It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
   
   It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
   
   If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-679204042


   /pending @maobaolong Do we have the same error with this patch as AWS?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-696453432


   Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
   
   It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
   
   It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
   
   If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-707967571


   @maobaolong 
   Once we are in similar behavior with AWS S3 we can get this in. I agree that NPE does not look right here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-709294091


   @bharatviswa504 Thanks for bring this PR back, as s3g without enable the security, so I think EndpointBase#SignatureProcessor shouldn't be inject as AWSSignatureProcessor or it should be a FakeSignatureProcessor,
   This can make the client without authorization setting can access s3g as expected.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-683519096


   /ready


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong edited a comment on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong edited a comment on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648490053


   @bharatviswa504 Thank you for your reply, can get the debug screenshot from the related Jira ticket, the result is complete auth header missing, and after i new the ~/.aws/credentials file on the goofys machine, and specify  the default profile by `goofys --profile default -o allow_other --file-mode=0777 --dir-mode=0777 --endpoint http://localhost:9878  user1 /data/home/micahzhao/testdir`
   
   the credentials file content is
   [default]
   aws_access_key_id=12345678
   aws_secret_access_key=12345678
   
   if i do that, the auth header will appear, and the NPE log will gone from the s3g log.
   
   So, you mean, we should keep s3g's behavior consistent with aws s3?, If so, we should throw the MALFORMED_HEADER exception and should catch it and just only log it.
   
   The following is the OzoneClientProducer#getClient. 
   
   ```java
   private OzoneClient getClient(OzoneConfiguration config) throws IOException {
       try {
         String awsAccessId = v4RequestParser.getAwsAccessId();
         UserGroupInformation remoteUser =
             UserGroupInformation.createRemoteUser(awsAccessId);
         if (OzoneSecurityUtil.isSecurityEnabled(config)) {
           LOG.debug("Creating s3 auth info for client.");
           try {
   
             OzoneTokenIdentifier identifier = new OzoneTokenIdentifier();
             identifier.setTokenType(S3AUTHINFO);
             identifier.setStrToSign(v4RequestParser.getStringToSign());
             identifier.setSignature(v4RequestParser.getSignature());
             identifier.setAwsAccessId(awsAccessId);
             identifier.setOwner(new Text(awsAccessId));
             if (LOG.isTraceEnabled()) {
               LOG.trace("Adding token for service:{}", omService);
             }
             Token<OzoneTokenIdentifier> token = new Token(identifier.getBytes(),
                 identifier.getSignature().getBytes(UTF_8),
                 identifier.getKind(),
                 omService);
             remoteUser.addToken(token);
           } catch (OS3Exception | URISyntaxException ex) {
             LOG.error("S3 auth info creation failed.");
             throw S3_AUTHINFO_CREATION_ERROR;
           }
   
         }
         UserGroupInformation.setLoginUser(remoteUser);
       } catch (Exception e) {
         LOG.error("Error: ", e);
       }
   
       if (omServiceID == null) {
         return OzoneClientFactory.getRpcClient(ozoneConfiguration);
       } else {
         // As in HA case, we need to pass om service ID.
         return OzoneClientFactory.getRpcClient(omServiceID, ozoneConfiguration);
       }
     }
   ```
   
   The inconsistent point is 
   
   ```java
    catch (Exception e) {
         LOG.error("Error: ", e);
       }
   ```
   
   It eat the exception only left a log.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-683519278


   /pending


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] dineshchitlangia commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
dineshchitlangia commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648216389


   @bharatviswa504 Could you review this? The change itself is very simple however I am not a fan of eliminating the PreCondition checks. Your thoughts are appreciated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-696453432


   Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
   
   It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
   
   It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
   
   If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648490053


   @bharatviswa504 Thank you for your reply, can get the debug screenshot from the related Jira ticket, the result is complete auth header missing, and after i new the ~/.aws/credentials file on the goofys machine, and specify  the default profile by `goofys --profile default -o allow_other --file-mode=0777 --dir-mode=0777 --endpoint http://localhost:9878  user1 /data/home/micahzhao/testdir`
   
   the credentials file content is
   [default]
   aws_access_key_id=12345678
   aws_secret_access_key=12345678
   
   if i do that, the auth header will appear
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-651683723


   > So, you mean, we should keep s3g's behavior consistent with aws s3?
   
   Yes, I think we should follow the behavior of AWS. There are so many s3 tools: doing exactly the same will cause less problem.
   
   But I agree that NPE should be replaced with more meaningful errors.  
   
   @maobaolong Do we have the same error with this patch as AWS?
   
   (@bharatviswa504 thanks the screenshot. I didn't know that postman can generate the aws signaure header. Very useful information, started to use postman again...)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648405855


   Thank You @maobaolong for the fix and @dineshchitlangia for tagging me.
   
   >Start a goofy without Authorization related setting.
   Can you elaborate this step what do you mean/how to set it up?
   
   I checked with the postman(curl tool), by empty access key/secret, the error I see from AWS S3 is as below.
   
   In the above step if you mean missing security access key/secret missing, then below is the error. (I have used postman, which generates auth header but missing access key
   ![Screen Shot 2020-06-23 at 1 40 37 PM](https://user-images.githubusercontent.com/8586345/85459546-21ad2900-b557-11ea-8c27-92a3115f2232.png)
   /secret)
   ```
   <?xml version="1.0" encoding="UTF-8"?>
   <Error>
       <Code>AuthorizationHeaderMalformed</Code>
       <Message>The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.</Message>
       <RequestId>64890549427133A8</RequestId>
       <HostId>rcJrOJhmw5P7JuDjs1wE/3GR7XlV7U54lSjtdxn7xBxdCX3wKyZ79ZIIJ30oF2R+CZ5PzOt90Rw=</HostId>
   </Error>
   ```
   
   If complete auth header missing, this is the error from AWS S3
   
   ```
   $ curl -X GET "https://b1-1990.s3.amazonaws.com/dir1/file1"
   <?xml version="1.0" encoding="UTF-8"?>
   <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>484B292350637469</RequestId><HostId>WKd5J5ytp/GOcindJU+60LNXtbxupzTFj2hjlEQzhTiwZBS3wN86BxvjT5ZBn9uh1eFqGHNMzNo=</HostId></Error>$
   ```
    


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] closed pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong edited a comment on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong edited a comment on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648490053


   @bharatviswa504 Thank you for your reply, can get the debug screenshot from the related Jira ticket, the result is complete auth header missing, and after i new the ~/.aws/credentials file on the goofys machine, and specify  the default profile by `goofys --profile default -o allow_other --file-mode=0777 --dir-mode=0777 --endpoint http://localhost:9878  user1 /data/home/micahzhao/testdir`
   
   the credentials file content is
   [default]
   aws_access_key_id=12345678
   aws_secret_access_key=12345678
   
   if i do that, the auth header will appear, and the NPE log will gone from the s3g log.
   
   So, you mean, we should keep s3g's behavior consistent with aws s3?, If so, we should throw the MALFORMED_HEADER exception and should catch it and just only log it.
   
   The following is the OzoneClientProducer#getClient. 
   
   ```java
   private OzoneClient getClient(OzoneConfiguration config) throws IOException {
       try {
         String awsAccessId = v4RequestParser.getAwsAccessId();
         UserGroupInformation remoteUser =
             UserGroupInformation.createRemoteUser(awsAccessId);
         if (OzoneSecurityUtil.isSecurityEnabled(config)) {
           LOG.debug("Creating s3 auth info for client.");
           try {
   
             OzoneTokenIdentifier identifier = new OzoneTokenIdentifier();
             identifier.setTokenType(S3AUTHINFO);
             identifier.setStrToSign(v4RequestParser.getStringToSign());
             identifier.setSignature(v4RequestParser.getSignature());
             identifier.setAwsAccessId(awsAccessId);
             identifier.setOwner(new Text(awsAccessId));
             if (LOG.isTraceEnabled()) {
               LOG.trace("Adding token for service:{}", omService);
             }
             Token<OzoneTokenIdentifier> token = new Token(identifier.getBytes(),
                 identifier.getSignature().getBytes(UTF_8),
                 identifier.getKind(),
                 omService);
             remoteUser.addToken(token);
           } catch (OS3Exception | URISyntaxException ex) {
             LOG.error("S3 auth info creation failed.");
             throw S3_AUTHINFO_CREATION_ERROR;
           }
   
         }
         UserGroupInformation.setLoginUser(remoteUser);
       } catch (Exception e) {
         LOG.error("Error: ", e);
       }
   
       if (omServiceID == null) {
         return OzoneClientFactory.getRpcClient(ozoneConfiguration);
       } else {
         // As in HA case, we need to pass om service ID.
         return OzoneClientFactory.getRpcClient(omServiceID, ozoneConfiguration);
       }
     }
   ```
   
   The inconsistent point is 
   
   ```java
    catch (Exception e) {
         LOG.error("Error: ", e);
       }
   ```
   
   It eat the exception only left a log.
   
   
   This is a similar issue which created by a Ozone developer. https://github.com/kahing/goofys/issues/513
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] maobaolong commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648662273


   @bharatviswa504 This issue is easy to reproduce by docker-compose.
   
   ```bash
   cd ./hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/ozone/
   export OZONE_REPLICATION_FACTOR=3
   ./run.sh
   
   cd ./hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT
   bin/ozone sh vol create s3v
   bin/ozone sh bucket create s3v/mybucket
   
    ./goofys --endpoint http://localhost:9878 mybucket /tmp/gfmp
   
   ll /tmp/gfmp
   
   docker logs ozone_s3g_1
   
   ```
   
   Full of NPE in the s3g log


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648405855


   Thank You @maobaolong for the fix and @dineshchitlangia for tagging me.
   
   @maobaolong 
   >Start a goofy without Authorization related setting.
   Can you elaborate this step what do you mean/how to set it up?
   
   I checked with the postman(curl tool), by empty access key/secret, the error I see from AWS S3 is as below.
   
   In the above step if you mean missing security access key/secret missing, then below is the error. (I have used postman, which generates auth header but missing access key
   ![Screen Shot 2020-06-23 at 1 40 37 PM](https://user-images.githubusercontent.com/8586345/85459546-21ad2900-b557-11ea-8c27-92a3115f2232.png)
   /secret)
   ```
   <?xml version="1.0" encoding="UTF-8"?>
   <Error>
       <Code>AuthorizationHeaderMalformed</Code>
       <Message>The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.</Message>
       <RequestId>64890549427133A8</RequestId>
       <HostId>rcJrOJhmw5P7JuDjs1wE/3GR7XlV7U54lSjtdxn7xBxdCX3wKyZ79ZIIJ30oF2R+CZ5PzOt90Rw=</HostId>
   </Error>
   ```
   
   If complete auth header missing, this is the error from AWS S3
   
   ```
   $ curl -X GET "https://b1-1990.s3.amazonaws.com/dir1/file1"
   <?xml version="1.0" encoding="UTF-8"?>
   <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>484B292350637469</RequestId><HostId>WKd5J5ytp/GOcindJU+60LNXtbxupzTFj2hjlEQzhTiwZBS3wN86BxvjT5ZBn9uh1eFqGHNMzNo=</HostId></Error>$
   ```
    


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] github-actions[bot] closed pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1110:
URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-707979758


   I see with injection, we see error in logs but we don't see proper error code at the S3 client.
   
   ![Screen Shot 2020-10-13 at 1 08 09 PM](https://user-images.githubusercontent.com/8586345/95910607-2ccdd400-0d55-11eb-9075-92a097f625fb.png)
   
   Not secure cluster, but tried with empty accesssKeyID and secret, still it is popping up with Bucket Not found. (Need to explore a bit how can we propagate this error)
   
   
   ```
   s3g_1       | 2020-10-13 19:56:56,215 [qtp1296456465-16] ERROR s3.OzoneClientProducer: Error:
   s3g_1       | org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws java.lang.Exception on org.apache.hadoop.ozone.s3.AWSSignatureProcessor@32f12a54
   s3g_1       | 	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99)
   s3g_1       | 	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80)
   s3g_1       | 	at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887)
   s3g_1       | 	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
   s3g_1       | 	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
   s3g_1       | 	at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99)
   s3g_1       | 	at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown Source)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68)
   s3g_1       | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   s3g_1       | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   s3g_1       | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   s3g_1       | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   s3g_1       | 	at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
   s3g_1       | 	at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
   s3g_1       | 	at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100)
   s3g_1       | 	at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161)
   s3g_1       | 	at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180)
   s3g_1       | 	at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
   s3g_1       | 	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785)
   s3g_1       | 	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885)
   s3g_1       | 	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
   s3g_1       | 	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358)
   s3g_1       | 	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369)
   s3g_1       | 	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
   s3g_1       | 	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
   s3g_1       | 	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
   s3g_1       | 	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873)
   s3g_1       | 	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159)
   s3g_1       | 	at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
   s3g_1       | 	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
   s3g_1       | 	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785)
   s3g_1       | 	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808)
   s3g_1       | 	at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61)
   s3g_1       | 	at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127)
   s3g_1       | 	at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70)
   s3g_1       | 	at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77)
   s3g_1       | 	at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153)
   s3g_1       | 	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
   s3g_1       | 	at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70)
   s3g_1       | 	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
   s3g_1       | 	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777)
   s3g_1       | 	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740)
   s3g_1       | 	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710)
   s3g_1       | 	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184)
   s3g_1       | 	at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54)
   s3g_1       | 	at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129)
   s3g_1       | 	at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93)
   s3g_1       | 	at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62)
   s3g_1       | 	at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
   s3g_1       | 	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
   s3g_1       | 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
   s3g_1       | 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
   s3g_1       | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
   s3g_1       | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
   s3g_1       | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
   s3g_1       | 	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
   s3g_1       | 	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
   s3g_1       | 	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
   s3g_1       | 	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
   s3g_1       | 	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
   s3g_1       | 	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
   s3g_1       | 	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
   s3g_1       | 	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
   s3g_1       | 	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   s3g_1       | 	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   s3g_1       | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
   s3g_1       | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
   s3g_1       | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   s3g_1       | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   s3g_1       | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   s3g_1       | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   s3g_1       | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
   s3g_1       | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   s3g_1       | 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   s3g_1       | 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   s3g_1       | 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   s3g_1       | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   s3g_1       | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   s3g_1       | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   s3g_1       | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   s3g_1       | 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
   s3g_1       | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   s3g_1       | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   s3g_1       | 	at java.base/java.lang.Thread.run(Thread.java:834)
   s3g_1       | Caused by: java.lang.reflect.InvocationTargetException
   s3g_1       | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   s3g_1       | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   s3g_1       | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   s3g_1       | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   s3g_1       | 	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:97)
   s3g_1       | 	... 117 more
   s3g_1       | Caused by: org.apache.hadoop.ozone.s3.exception.OS3Exception
   s3g_1       | 	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:112)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV4.validateCredentials(AuthorizationHeaderV4.java:163)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV4.parseAuthHeader(AuthorizationHeaderV4.java:101)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV4.<init>(AuthorizationHeaderV4.java:70)
   s3g_1       | 	at org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init(AWSSignatureProcessor.java:124)
   s3g_1       | 	... 122 more
   s3g_1       | 2020-10-13 19:56:56,239 [qtp1296456465-16] ERROR exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
   s3g_1       | <Error>
   s3g_1       |   <Code>NoSuchBucket</Code>
   s3g_1       |   <Message>The specified bucket does not exist</Message>
   s3g_1       |   <Resource>b12345</Resource>
   s3g_1       |   <RequestId/>
   s3g_1       | </Error>
   s3g_1       |
   ```
   
   @maobaolong  @elek Any thoughts/suggestions?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org