You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/11/25 14:57:27 UTC

[GitHub] [ozone] elek opened a new pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

elek opened a new pull request #1628:
URL: https://github.com/apache/ozone/pull/1628


   ## What changes were proposed in this pull request?
   
   AWS S3 accepts authorization information both from headers and query parameter.
   
   Ozone s3g implementation parses only the headers.
   
   This patch makes introduces the query parameter based authentication support with small reorganization of the existing logic to make it possible to extend it in this way.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4506
   
   ## How was this patch tested?
   
   rclone uses query parameter based authentication. 
   
   `rclone copy `pwd` ozone:bucket1/` failed without this patch, but after this patch it worked well.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538907902



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  private static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,

Review comment:
       Can we combine this method and below.
   
   And also can we call this method like createStringToSign, it is confusing as it says createSignatureBase IMO.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538920466



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  private static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        context.getUriInfo().getQueryParameters());
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      Map<String, String> headers,
+      MultivaluedMap<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope, canonicalRequest;
+    credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(headers.get(X_AMAZ_DATE) + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams);
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+    }
+
+    LOG.debug("StringToSign:[{}]", strToSign);

Review comment:
       Minor: Move this also inside isDebugEnabled, here it is string, I believe we don't require LOG.isDebugEnabled() right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r603895046



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AWSSignatureProcessor.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.enterprise.context.RequestScoped;
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MultivaluedMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.ozone.s3.HeaderPreprocessor;
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.SignatureInfo.Version;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Parser to process AWS V2 & V4 auth request. Creates string to sign and auth
+ * header. For more details refer to AWS documentation https://docs.aws
+ * .amazon.com/general/latest/gr/sigv4-create-canonical-request.html.
+ **/
+@RequestScoped
+public class AWSSignatureProcessor implements SignatureProcessor {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(AWSSignatureProcessor.class);
+
+  @Context
+  private ContainerRequestContext context;
+
+  public SignatureInfo parseSignature() throws OS3Exception {
+
+    LowerCaseKeyStringMap headers =

Review comment:
       Good question, but I am not sure. As far as I see it's added in #1098 I think I just kept the same logic what we used before...




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek merged pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek merged pull request #1628:
URL: https://github.com/apache/ozone/pull/1628


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r605552666



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")
+          .replaceAll("%7E", "~");
+    } catch (UnsupportedEncodingException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static String getQueryParamString(
+      Map<String, String> queryMap
+  ) {
+    List<String> params = new ArrayList<>(queryMap.keySet());
+
+    // Sort by name, then by value
+    Collections.sort(params, (o1, o2) -> o1.equals(o2) ?
+        queryMap.get(o1).compareTo(queryMap.get(o2)) :
+        o1.compareTo(o2));
+
+    StringBuilder result = new StringBuilder();
+    for (String p : params) {
+      if (!p.equals("X-Amz-Signature")) {

Review comment:
       Question: This check is not there in the old code, newly added. Any reason for this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-775023594


   @bharatviswa504 All of the comments are addressed, and I added more unit tests to test proper working (and X-Amz-Data handling).
   
   Can you please and other look when you have time?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r605554470



##########
File path: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/signature/TestStringToSignProducer.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.s3.HeaderPreprocessor;
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test string2sign creation.
+ */
+public class TestStringToSignProducer {
+
+  @Test
+  public void test() throws Exception {
+
+    LowerCaseKeyStringMap headers = new LowerCaseKeyStringMap();
+    headers.put("Content-Length", "123");
+    headers.put("Host", "0.0.0.0:9878");
+    headers.put("X-AMZ-Content-Sha256", "Content-SHA");
+    headers.put("X-AMZ-Date", "123");
+    headers.put("Content-Type", "ozone/mpu");
+    headers.put(HeaderPreprocessor.ORIGINAL_CONTENT_TYPE, "streaming");
+
+    String authHeader =
+        "AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1"
+            + "/s3/aws4_request, "
+            + "SignedHeaders=host;x-amz-content-sha256;x-amz-date;"
+            + "content-type, "
+            + "Signature"
+            +
+            "=db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2";
+
+    headers.put("Authorization",
+        authHeader);
+
+    Map<String, String> queryParameters = new HashMap<>();
+
+    final SignatureInfo signatureInfo =
+        new AuthorizationV4HeaderParser(authHeader, "123") {
+          @Override
+          public void validateDateRange(Credential credentialObj)
+              throws OS3Exception {
+            //NOOP
+          }
+        }.parseSignature();
+
+    headers.fixContentType();
+
+    final String signatureBase =
+        StringToSignProducer.createSignatureBase(
+            signatureInfo,
+            "http",
+            "GET",
+            "/buckets",
+            headers,
+            queryParameters);
+
+    Assert.assertEquals(
+        "String to sign is invalid",

Review comment:
       Sorry for not being clear, my question is how you have got the siggnatureBase computed. I see in other test it is mentioned it is from AWS documentation, is this also from there?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-809300747


   Can we also add a docker test to test rclone tool?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-809299999


   Thank You @elek for clear separation between Authorization Header Parsing and Sign generation.
   Over all LGTM, few minor comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r599473226



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AWSSignatureProcessor.java
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.enterprise.context.RequestScoped;
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MultivaluedMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.ozone.s3.HeaderPreprocessor;
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.SignatureInfo.Version;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Parser to process AWS V2 & V4 auth request. Creates string to sign and auth
+ * header. For more details refer to AWS documentation https://docs.aws
+ * .amazon.com/general/latest/gr/sigv4-create-canonical-request.html.
+ **/
+@RequestScoped
+public class AWSSignatureProcessor implements SignatureProcessor {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(AWSSignatureProcessor.class);
+
+  @Context
+  private ContainerRequestContext context;
+
+  public SignatureInfo parseSignature() throws OS3Exception {
+
+    LowerCaseKeyStringMap headers =

Review comment:
       Question: why do we convert header keys to lowercase?

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -125,7 +127,7 @@ public static String createSignatureBase(
         headers,
         queryParams,
         !signatureInfo.isSignPayload());
-
+    System.out.println(canonicalRequest);

Review comment:
       Can we do Log here?
   I see it is already logged when debug enabled. do we want to remove this?

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
##########
@@ -138,6 +146,21 @@ private OzoneClient getClient(OzoneConfiguration config)
     return ozoneClient;
   }
 
+  @NotNull
+  @VisibleForTesting
+  OzoneClient createOzoneClient() throws IOException {
+    if (omServiceID == null) {
+      return OzoneClientFactory.getRpcClient(ozoneConfiguration);
+    } else {
+      // As in HA case, we need to pass om service ID.
+      return OzoneClientFactory.getRpcClient(omServiceID,
+          ozoneConfiguration);
+    }
+  }
+
+  private void getSignatureInfo() {

Review comment:
       Minor: Unused method.

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    System.out.println(canonicalRequest);
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")

Review comment:
       Why are we doing this?
   
   
   https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
   Here there is an example urlEncode method is given, don't see such replace here?

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java
##########
@@ -57,6 +57,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+

Review comment:
       Minor: Unnecessary change

##########
File path: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/signature/TestStringToSignProducer.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.s3.HeaderPreprocessor;
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test string2sign creation.
+ */
+public class TestStringToSignProducer {
+
+  @Test
+  public void test() throws Exception {
+
+    LowerCaseKeyStringMap headers = new LowerCaseKeyStringMap();
+    headers.put("Content-Length", "123");
+    headers.put("Host", "0.0.0.0:9878");
+    headers.put("X-AMZ-Content-Sha256", "Content-SHA");
+    headers.put("X-AMZ-Date", "123");
+    headers.put("Content-Type", "ozone/mpu");
+    headers.put(HeaderPreprocessor.ORIGINAL_CONTENT_TYPE, "streaming");
+
+    String authHeader =
+        "AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1"
+            + "/s3/aws4_request, "
+            + "SignedHeaders=host;x-amz-content-sha256;x-amz-date;"
+            + "content-type, "
+            + "Signature"
+            +
+            "=db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2";
+
+    headers.put("Authorization",
+        authHeader);
+
+    Map<String, String> queryParameters = new HashMap<>();
+
+    final SignatureInfo signatureInfo =
+        new AuthorizationV4HeaderParser(authHeader, "123") {
+          @Override
+          public void validateDateRange(Credential credentialObj)
+              throws OS3Exception {
+            //NOOP
+          }
+        }.parseSignature();
+
+    headers.fixContentType();
+
+    final String signatureBase =
+        StringToSignProducer.createSignatureBase(
+            signatureInfo,
+            "http",
+            "GET",
+            "/buckets",
+            headers,
+            queryParameters);
+
+    Assert.assertEquals(
+        "String to sign is invalid",

Review comment:
       Not understood here message is String to sign is invalid
   But signatureBase is matching with signature.
   Could you explain how/what is being tested here and how the value is generated




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-810039816


   Thanks a lot the review @bharatviswa504 
   
   I pushed a commit with the minor fixes.
   
   > Can we also add a docker test to test rclone tool?
   
   Good idea, I created, https://issues.apache.org/jira/browse/HDDS-5045 to track it. (We need to update the ozone-runner docker image first with rclone)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538898613



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AuthorizationV4HeaderParser.java
##########
@@ -1,86 +1,78 @@
 /**
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with this
- * work for additional information regarding copyright ownership.  The ASF
- * licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
  * <p>
  * http://www.apache.org/licenses/LICENSE-2.0
  * <p>
  * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
  */
+package org.apache.hadoop.ozone.s3.signature;
 
-package org.apache.hadoop.ozone.s3.header;
-
+import java.time.LocalDate;
+import java.util.Collection;
 
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import org.apache.commons.codec.DecoderException;
-import org.apache.commons.codec.binary.Hex;
 import org.apache.hadoop.ozone.s3.exception.OS3Exception;
 import org.apache.hadoop.ozone.s3.exception.S3ErrorTable;
+import org.apache.hadoop.ozone.s3.signature.SignatureInfo.Version;
 import org.apache.hadoop.util.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.time.LocalDate;
-import java.util.Collection;
 
+import com.google.common.annotations.VisibleForTesting;
 import static java.time.temporal.ChronoUnit.DAYS;
+import org.apache.commons.codec.DecoderException;
+import org.apache.commons.codec.binary.Hex;
 import static org.apache.commons.lang3.StringUtils.isEmpty;
 import static org.apache.commons.lang3.StringUtils.isNotEmpty;
 import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.MALFORMED_HEADER;
-import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.AWS4_SIGNING_ALGORITHM;
-import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.DATE_FORMATTER;
+import static org.apache.hadoop.ozone.s3.signature.SignatureProcessor.AWS4_SIGNING_ALGORITHM;
+import static org.apache.hadoop.ozone.s3.signature.SignatureProcessor.DATE_FORMATTER;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
- * S3 Authorization header.
- * Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using
- * -authorization-header.html
+ * Class to parse v4 auth information from header.
  */
-public class AuthorizationHeaderV4 {
+public class AuthorizationV4HeaderParser implements SignatureParser {
 
-  private final static Logger LOG = LoggerFactory.getLogger(
-      AuthorizationHeaderV4.class);
+  private static final Logger LOG =
+      LoggerFactory.getLogger(AuthorizationV4HeaderParser.class);
 
   private final static String CREDENTIAL = "Credential=";
   private final static String SIGNEDHEADERS = "SignedHeaders=";
   private final static String SIGNATURE = "Signature=";
 
   private String authHeader;
-  private String algorithm;
-  private String credential;
-  private String signedHeadersStr;
-  private String signature;
-  private Credential credentialObj;
-  private Collection<String> signedHeaders;
 
-  /**
-   * Construct AuthorizationHeader object.
-   * @param header
-   */
-  public AuthorizationHeaderV4(String header) throws OS3Exception {
-    Preconditions.checkNotNull(header);
-    this.authHeader = header;
-    parseAuthHeader();
+  public AuthorizationV4HeaderParser(String authHeader) {
+    this.authHeader = authHeader;
   }
 
   /**
    * This method parses authorization header.
+   * <p>
+   * Authorization Header sample:
+   * AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1/s3
+   * /aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date,
+   * Signature
+   * =db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2
    *
-   *  Authorization Header sample:
-   *  AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1/s3
-   *  /aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date,
-   * Signature=db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2
    * @throws OS3Exception
    */
   @SuppressWarnings("StringSplitter")
-  public void parseAuthHeader() throws OS3Exception {
+  @Override
+  public SignatureInfo parseSignature() throws OS3Exception {
+    if (authHeader == null || authHeader
+        .startsWith(AuthorizationV2HeaderParser.IDENTIFIER + " ")) {

Review comment:
       V2 starts with AWS, V4 Starts with AWS4




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-809107660


   > Unknown patch can make
   > the review process slow
   > but if you need more info
   > please: just let me know
   
   cc @bharatviswa504 @prashantpogde


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-784730196


   @elek 
   Sorry for the late response.
   I am a little busy with other tasks, will get to it by next week. If others can help in the review, pls proceed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-803830503


   > Roses are red
   > Violets are blue
   > Eventually -- I hope
   > Will get a review...
   
   cc @bharatviswa504 @prashantpogde 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r539174358



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  private static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,

Review comment:
       Sure, we can. But I kept them separated to make them more testable with separating the logic to extract information from the logic to create the sign.
   
   I added the `@VisibleForTesting` annotation and simplified the test (mocks are removed) to show this concept, but I am open to merge it if you don't like it.
   
   (See 48676ea0b)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r605553643



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    System.out.println(canonicalRequest);
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")

Review comment:
       Yes, this is completely old code. I just see one new change.
   
   >We can try to clean up it, but I would prefer to do it outside of this patch (just to make clear what is related to the >simple reorganization and what is related to a logic change...)
   Agreed, already this is a big change, no need of further refactor.
   

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    System.out.println(canonicalRequest);
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")

Review comment:
       Yes, this is completely old code. I just see one new change.
   
   >We can try to clean up it, but I would prefer to do it outside of this patch (just to make clear what is related to the >simple reorganization and what is related to a logic change...)
   
   Agreed, already this is a big change, no need of further refactor.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r539177249



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
##########
@@ -130,14 +139,29 @@ private OzoneClient getClient(OzoneConfiguration config)
     } catch (Throwable t) {
       // For any other critical errors during object creation throw Internal
       // error.
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Error during Client Creation: ", t);
-      }
+      //      if (LOG.isDebugEnabled()) {

Review comment:
       Good, question. I am thinking what supposed to be the right error handling.
   
   AFAIK all of these exceptions are related to the user / input errors. They are not really application errors. User errors are not required to be logged, IMHO a permission denied is part of the normal business.
   
   I would restore the `LOG.isDebugEnabled()` here, if you agree...  (8fe1202da) 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538915447



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
##########
@@ -130,14 +139,29 @@ private OzoneClient getClient(OzoneConfiguration config)
     } catch (Throwable t) {
       // For any other critical errors during object creation throw Internal
       // error.
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Error during Client Creation: ", t);
-      }
+      //      if (LOG.isDebugEnabled()) {

Review comment:
       Here the debug is removed, do we want to add the same when OS3Exception also?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-804077585


   > > Roses are red
   > > Violets are blue
   > > Eventually -- I hope
   > > Will get a review...
   > 
   > cc @bharatviswa504 @prashantpogde
   
   @elek  Sorry for the delay, got stuck with other tasks. I will get to this tomorrow for sure, as no one got to review this. Hope this helps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-782159681


   @bharatviswa504 do you have time to check it by any chance?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r605664555



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")
+          .replaceAll("%7E", "~");
+    } catch (UnsupportedEncodingException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static String getQueryParamString(
+      Map<String, String> queryMap
+  ) {
+    List<String> params = new ArrayList<>(queryMap.keySet());
+
+    // Sort by name, then by value
+    Collections.sort(params, (o1, o2) -> o1.equals(o2) ?
+        queryMap.get(o1).compareTo(queryMap.get(o2)) :
+        o1.compareTo(o2));
+
+    StringBuilder result = new StringBuilder();
+    for (String p : params) {
+      if (!p.equals("X-Amz-Signature")) {

Review comment:
       This is the code which creates the canonical string of the query parameters to sign it later.
   
   If the signature itself is added as a query parameter, it should be excluded from the string itself (to have a predictable string to sign). 
   
   TLDR: earlier we didn't support `X-Amz-Signature` query parameter therefore we ignored this exclusion.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r603901111



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -125,7 +127,7 @@ public static String createSignatureBase(
         headers,
         queryParams,
         !signatureInfo.isSignPayload());
-
+    System.out.println(canonicalRequest);

Review comment:
       Yup, thanks. Removed.

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java
##########
@@ -57,6 +57,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+

Review comment:
       Thanks. reverted.

##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
##########
@@ -138,6 +146,21 @@ private OzoneClient getClient(OzoneConfiguration config)
     return ozoneClient;
   }
 
+  @NotNull
+  @VisibleForTesting
+  OzoneClient createOzoneClient() throws IOException {
+    if (omServiceID == null) {
+      return OzoneClientFactory.getRpcClient(ozoneConfiguration);
+    } else {
+      // As in HA case, we need to pass om service ID.
+      return OzoneClientFactory.getRpcClient(omServiceID,
+          ozoneConfiguration);
+    }
+  }
+
+  private void getSignatureInfo() {

Review comment:
       Thanks, removed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-812073167


   > Roses are reds
   > Violest are blue
   > This pr is merged
   > Thanks @bharatviswa504 to you
   
   :smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-785836560


   >  I am a little busy with other tasks, will get to it by next week.
   
   Thanks a lot. Happy to give more information / explanation if you need...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek edited a comment on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek edited a comment on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-812073167


   > Roses are red
   > Violets are blue
   > This pr is merged
   > Thanks @bharatviswa504 to you
   
   :smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538896778



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AuthorizationV4QueryParser.java
##########
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.core.MultivaluedMap;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.SignatureInfo.Version;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Parser for getting auth info from query parameters.
+ * <p>
+ * See: https://docs.aws.amazon
+ * .com/AmazonS3/latest/API/sigv4-query-string-auth.html
+ */
+public class AuthorizationV4QueryParser implements SignatureParser {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(AuthorizationV4QueryParser.class);
+
+  private final MultivaluedMap<String, String> queryParameters;
+
+  public AuthorizationV4QueryParser(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    this.queryParameters = queryParameters;
+  }
+
+  @Override
+  public SignatureInfo parseSignature() throws OS3Exception {
+
+    if (!queryParameters.containsKey("X-Amz-Signature")) {
+      return null;
+    }
+
+    Credential credential =
+        new Credential(queryParameters.getFirst("X-Amz-Credential"));
+
+    return new SignatureInfo(
+        Version.V4,
+        queryParameters.getFirst("X-Amz-Date"),
+        credential.getAccessKeyID(),

Review comment:
       https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
   This has X-Amz-Expires, we are not considering this param




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r603900472



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import javax.ws.rs.container.ContainerRequestContext;
+import javax.ws.rs.core.MultivaluedMap;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.net.UnknownHostException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+import org.apache.hadoop.util.StringUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import static java.time.temporal.ChronoUnit.SECONDS;
+import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.S3_AUTHINFO_CREATION_ERROR;
+import org.apache.kerby.util.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Stateless utility to create stringToSign, the base of the signature.
+ */
+public final class StringToSignProducer {
+
+  public static final String X_AMZ_CONTENT_SHA256 = "X-Amz-Content-SHA256";
+  public static final String X_AMAZ_DATE = "X-Amz-Date";
+  private static final Logger LOG =
+      LoggerFactory.getLogger(StringToSignProducer.class);
+  private static final Charset UTF_8 = StandardCharsets.UTF_8;
+  private static final String NEWLINE = "\n";
+  private static final String HOST = "host";
+  private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
+  /**
+   * Seconds in a week, which is the max expiration time Sig-v4 accepts.
+   */
+  private static final long PRESIGN_URL_MAX_EXPIRATION_SECONDS =
+      60 * 60 * 24 * 7;
+  public static final DateTimeFormatter TIME_FORMATTER =
+      DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")
+          .withZone(ZoneOffset.UTC);
+
+  private StringToSignProducer() {
+  }
+
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      ContainerRequestContext context
+  ) throws Exception {
+    return createSignatureBase(signatureInfo,
+        context.getUriInfo().getRequestUri().getScheme(),
+        context.getMethod(),
+        context.getUriInfo().getRequestUri().getPath(),
+        LowerCaseKeyStringMap.fromHeaderMap(context.getHeaders()),
+        fromMultiValueToSingleValueMap(
+            context.getUriInfo().getQueryParameters()));
+  }
+
+  @VisibleForTesting
+  public static String createSignatureBase(
+      SignatureInfo signatureInfo,
+      String scheme,
+      String method,
+      String uri,
+      LowerCaseKeyStringMap headers,
+      Map<String, String> queryParams
+  ) throws Exception {
+    StringBuilder strToSign = new StringBuilder();
+    // According to AWS sigv4 documentation, authorization header should be
+    // in following format.
+    // Authorization: algorithm Credential=access key ID/credential scope,
+    // SignedHeaders=SignedHeaders, Signature=signature
+
+    // Construct String to sign in below format.
+    // StringToSign =
+    //    Algorithm + \n +
+    //    RequestDateTime + \n +
+    //    CredentialScope + \n +
+    //    HashedCanonicalRequest
+    String credentialScope = signatureInfo.getCredentialScope();
+
+    // If the absolute path is empty, use a forward slash (/)
+    uri = (uri.trim().length() > 0) ? uri : "/";
+    // Encode URI and preserve forward slashes
+    strToSign.append(signatureInfo.getAlgorithm() + NEWLINE);
+    strToSign.append(signatureInfo.getDateTime() + NEWLINE);
+    strToSign.append(credentialScope + NEWLINE);
+
+    String canonicalRequest = buildCanonicalRequest(
+        scheme,
+        method,
+        uri,
+        signatureInfo.getSignedHeaders(),
+        headers,
+        queryParams,
+        !signatureInfo.isSignPayload());
+    System.out.println(canonicalRequest);
+    strToSign.append(hash(canonicalRequest));
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("canonicalRequest:[{}]", canonicalRequest);
+      LOG.debug("StringToSign:[{}]", strToSign);
+    }
+
+    return strToSign.toString();
+  }
+
+  public static Map<String, String> fromMultiValueToSingleValueMap(
+      MultivaluedMap<String, String> queryParameters
+  ) {
+    Map<String, String> result = new HashMap<>();
+    for (String key : queryParameters.keySet()) {
+      result.put(key, queryParameters.getFirst(key));
+    }
+    return result;
+  }
+
+  public static String hash(String payload) throws NoSuchAlgorithmException {
+    MessageDigest md = MessageDigest.getInstance("SHA-256");
+    md.update(payload.getBytes(UTF_8));
+    return Hex.encode(md.digest()).toLowerCase();
+  }
+
+  @VisibleForTesting
+  public static String buildCanonicalRequest(
+      String schema,
+      String method,
+      String uri,
+      String signedHeaders,
+      Map<String, String> headers,
+      Map<String, String> queryParams,
+      boolean unsignedPayload
+  ) throws OS3Exception {
+
+    Iterable<String> parts = split("/", uri);
+    List<String> encParts = new ArrayList<>();
+    for (String p : parts) {
+      encParts.add(urlEncode(p));
+    }
+    String canonicalUri = join("/", encParts);
+
+    String canonicalQueryStr = getQueryParamString(queryParams);
+
+    StringBuilder canonicalHeaders = new StringBuilder();
+
+    for (String header : StringUtils.getStringCollection(signedHeaders, ";")) {
+      canonicalHeaders.append(header.toLowerCase());
+      canonicalHeaders.append(":");
+      if (headers.containsKey(header)) {
+        String headerValue = headers.get(header);
+        canonicalHeaders.append(headerValue);
+        canonicalHeaders.append(NEWLINE);
+
+        // Set for testing purpose only to skip date and host validation.
+        validateSignedHeader(schema, header, headerValue);
+
+      } else {
+        throw new RuntimeException("Header " + header + " not present in " +
+            "request but requested to be signed.");
+      }
+    }
+
+    String payloadHash;
+    if (UNSIGNED_PAYLOAD.equals(
+        headers.get(X_AMZ_CONTENT_SHA256)) || unsignedPayload) {
+      payloadHash = UNSIGNED_PAYLOAD;
+    } else {
+      payloadHash = headers.get(X_AMZ_CONTENT_SHA256);
+    }
+    String canonicalRequest = method + NEWLINE
+        + canonicalUri + NEWLINE
+        + canonicalQueryStr + NEWLINE
+        + canonicalHeaders + NEWLINE
+        + signedHeaders + NEWLINE
+        + payloadHash;
+    return canonicalRequest;
+  }
+
+  /**
+   * String join that also works with empty strings.
+   *
+   * @return joined string
+   */
+  private static String join(String glue, List<String> parts) {
+    StringBuilder result = new StringBuilder();
+    boolean addSeparator = false;
+    for (String p : parts) {
+      if (addSeparator) {
+        result.append(glue);
+      }
+      result.append(p);
+      addSeparator = true;
+    }
+    return result.toString();
+  }
+
+  /**
+   * Returns matching strings.
+   *
+   * @param regex Regular expression to split by
+   * @param whole The string to split
+   * @return pieces
+   */
+  private static Iterable<String> split(String regex, String whole) {
+    Pattern p = Pattern.compile(regex);
+    Matcher m = p.matcher(whole);
+    List<String> result = new ArrayList<>();
+    int pos = 0;
+    while (m.find()) {
+      result.add(whole.substring(pos, m.start()));
+      pos = m.end();
+    }
+    result.add(whole.substring(pos));
+    return result;
+  }
+
+  private static String urlEncode(String str) {
+    try {
+
+      return URLEncoder.encode(str, UTF_8.name())
+          .replaceAll("\\+", "%20")

Review comment:
       Again, it's a good question, but I don't know exactly. I didn't modify the logic in this patch.
   
   It was added by Ajay in apache/hadoop#561. I tried to re-organize the URLEncoder, but tests started to fail so I decided to keep all the existing logic as is.
   
   We can try to clean up it, but I would prefer to do it outside of this patch (just to make clear what is related to the simple reorganization and what is related to a logic change...)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r603896921



##########
File path: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/signature/TestStringToSignProducer.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.s3.signature;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.s3.HeaderPreprocessor;
+import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.LowerCaseKeyStringMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test string2sign creation.
+ */
+public class TestStringToSignProducer {
+
+  @Test
+  public void test() throws Exception {
+
+    LowerCaseKeyStringMap headers = new LowerCaseKeyStringMap();
+    headers.put("Content-Length", "123");
+    headers.put("Host", "0.0.0.0:9878");
+    headers.put("X-AMZ-Content-Sha256", "Content-SHA");
+    headers.put("X-AMZ-Date", "123");
+    headers.put("Content-Type", "ozone/mpu");
+    headers.put(HeaderPreprocessor.ORIGINAL_CONTENT_TYPE, "streaming");
+
+    String authHeader =
+        "AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1"
+            + "/s3/aws4_request, "
+            + "SignedHeaders=host;x-amz-content-sha256;x-amz-date;"
+            + "content-type, "
+            + "Signature"
+            +
+            "=db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2";
+
+    headers.put("Authorization",
+        authHeader);
+
+    Map<String, String> queryParameters = new HashMap<>();
+
+    final SignatureInfo signatureInfo =
+        new AuthorizationV4HeaderParser(authHeader, "123") {
+          @Override
+          public void validateDateRange(Credential credentialObj)
+              throws OS3Exception {
+            //NOOP
+          }
+        }.parseSignature();
+
+    headers.fixContentType();
+
+    final String signatureBase =
+        StringToSignProducer.createSignatureBase(
+            signatureInfo,
+            "http",
+            "GET",
+            "/buckets",
+            headers,
+            queryParameters);
+
+    Assert.assertEquals(
+        "String to sign is invalid",

Review comment:
       I may misunderstand the question, but the message is printed out when the two values are **not** equal:
   
   From JUnit javadoc: 
   
   > Asserts that two objects are equal. If they are not, an {@link AssertionError} is thrown with the given message.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#discussion_r538898372



##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AuthorizationV4HeaderParser.java
##########
@@ -1,86 +1,78 @@
 /**
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with this
- * work for additional information regarding copyright ownership.  The ASF
- * licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
  * <p>
  * http://www.apache.org/licenses/LICENSE-2.0
  * <p>
  * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
  */
+package org.apache.hadoop.ozone.s3.signature;
 
-package org.apache.hadoop.ozone.s3.header;
-
+import java.time.LocalDate;
+import java.util.Collection;
 
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import org.apache.commons.codec.DecoderException;
-import org.apache.commons.codec.binary.Hex;
 import org.apache.hadoop.ozone.s3.exception.OS3Exception;
 import org.apache.hadoop.ozone.s3.exception.S3ErrorTable;
+import org.apache.hadoop.ozone.s3.signature.SignatureInfo.Version;
 import org.apache.hadoop.util.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.time.LocalDate;
-import java.util.Collection;
 
+import com.google.common.annotations.VisibleForTesting;
 import static java.time.temporal.ChronoUnit.DAYS;
+import org.apache.commons.codec.DecoderException;
+import org.apache.commons.codec.binary.Hex;
 import static org.apache.commons.lang3.StringUtils.isEmpty;
 import static org.apache.commons.lang3.StringUtils.isNotEmpty;
 import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.MALFORMED_HEADER;
-import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.AWS4_SIGNING_ALGORITHM;
-import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.DATE_FORMATTER;
+import static org.apache.hadoop.ozone.s3.signature.SignatureProcessor.AWS4_SIGNING_ALGORITHM;
+import static org.apache.hadoop.ozone.s3.signature.SignatureProcessor.DATE_FORMATTER;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
- * S3 Authorization header.
- * Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using
- * -authorization-header.html
+ * Class to parse v4 auth information from header.
  */
-public class AuthorizationHeaderV4 {
+public class AuthorizationV4HeaderParser implements SignatureParser {
 
-  private final static Logger LOG = LoggerFactory.getLogger(
-      AuthorizationHeaderV4.class);
+  private static final Logger LOG =
+      LoggerFactory.getLogger(AuthorizationV4HeaderParser.class);
 
   private final static String CREDENTIAL = "Credential=";
   private final static String SIGNEDHEADERS = "SignedHeaders=";
   private final static String SIGNATURE = "Signature=";
 
   private String authHeader;
-  private String algorithm;
-  private String credential;
-  private String signedHeadersStr;
-  private String signature;
-  private Credential credentialObj;
-  private Collection<String> signedHeaders;
 
-  /**
-   * Construct AuthorizationHeader object.
-   * @param header
-   */
-  public AuthorizationHeaderV4(String header) throws OS3Exception {
-    Preconditions.checkNotNull(header);
-    this.authHeader = header;
-    parseAuthHeader();
+  public AuthorizationV4HeaderParser(String authHeader) {
+    this.authHeader = authHeader;
   }
 
   /**
    * This method parses authorization header.
+   * <p>
+   * Authorization Header sample:
+   * AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1/s3
+   * /aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date,
+   * Signature
+   * =db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2
    *
-   *  Authorization Header sample:
-   *  AWS4-HMAC-SHA256 Credential=AKIAJWFJK62WUTKNFJJA/20181009/us-east-1/s3
-   *  /aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date,
-   * Signature=db81b057718d7c1b3b8dffa29933099551c51d787b3b13b9e0f9ebed45982bf2
    * @throws OS3Exception
    */
   @SuppressWarnings("StringSplitter")
-  public void parseAuthHeader() throws OS3Exception {
+  @Override
+  public SignatureInfo parseSignature() throws OS3Exception {
+    if (authHeader == null || authHeader
+        .startsWith(AuthorizationV2HeaderParser.IDENTIFIER + " ")) {

Review comment:
       AuthorizationV4HeaderParser.IDENTIFIER=AWS4




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-800033861


   @bharatviswa504 @prashantpogde Did you have any time to check it by any chance? Happy to help to explain if it helps...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 edited a comment on pull request #1628: HDDS-4506. Support query parameter based v4 auth in S3g

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #1628:
URL: https://github.com/apache/ozone/pull/1628#issuecomment-804077585


   > > Roses are red
   > > Violets are blue
   > > Eventually -- I hope
   > > Will get a review...
   > 
   > cc @bharatviswa504 @prashantpogde
   
   @elek  Sorry for the delay, got stuck with other tasks, now they seem to be under control. I will get to this tomorrow for sure, as no one got to review this. Hope this helps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org