[jira] [Created] (HDDS-7992) Copy and move command for MPU objects from awscli v2 not working

["UENISHI Kota (Jira)" <ji...@apache.org>]

UENISHI Kota created HDDS-7992:
----------------------------------

             Summary: Copy and move command for MPU objects from awscli v2 not working
                 Key: HDDS-7992
                 URL: https://issues.apache.org/jira/browse/HDDS-7992
             Project: Apache Ozone
          Issue Type: Bug
          Components: S3
    Affects Versions: 1.2.1, 1.3.0
         Environment: * Affected AWSCLI version: \{{aws-cli/2.10.1 Python/3.9.11 Linux/6.1.12-arch1-1 docker/x86_64.amzn.2 prompt/off}}
 * It does not happen for AWSCLI v1. So our workaround is to use it for now.
 * We use Ubuntu Linux 22.04 for client side
 * We have two clusters Running Ozone 1.3, one runs Ubuntu 18.04 and the other runs 22.04.
            Reporter: UENISHI Kota


AWSCLI v2 running {{aws s3 --endpoint=[https://our-endpoint.example.com/] cp s3://some/object s3://some/destination}} fails with returning huge error message printing the object itself.

The {{--debug}} log indicates that awscli v2 sends out weird HTTP request beforehand:

{{
2023-02-21 11:15:17,570 - ThreadPoolExecutor-1_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://jm01z2cs10.s.nf10.net/kota/32MiB?tagging, headers={'User-Agent': b'aws-cli/2.9.15 Python/3.9.11 Linux/5.15.0-56-generic exe/x86_64.ubuntu.22 prompt/off command/s3.cp', 'X-Amz-Date': b'20230221T021517Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=kota@PFN.IO/20230221/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=dd3b6b6e19776e123c81c699b1f45825507857c19e817d5a823fca3aa7361aea'}>
2023-02-21 11:15:17,571 - ThreadPoolExecutor-1_0 - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.9.15/dist/awscli/botocore/cacert.pem
2023-02-21 11:15:17,593 - ThreadPoolExecutor-1_0 - urllib3.connectionpool - DEBUG - https://jm01z2cs10.s.nf10.net:443 "GET /kota/32MiB?tagging HTTP/1.1" 200 33554432
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG - Response headers: {'Date': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Cache-Control': 'no-cache', 'Expires': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Pragma': 'no-cache', 'Content-Type': 'application/octet-stream', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-FRAME-OPTIONS': 'SAMEORIGIN', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Tue, 21 Feb 2023 01:57:51 GMT', 'Server': 'Ozone', 'x-amz-id-2': '5Q3e2Cl7', 'x-amz-request-id': 'e8c7b5a0-d0fe-4cfa-8508-abd34574c793', 'Content-Length': '33554432'}
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG - Response body:
b'tm\x1dv3\xf7\x880=\x94\x17\x14\xd1Y\xa7\xa3\x11\xa8\x04%\xa8\xb2\x19Q\x98G\xde\xca/,\x04\x0e\xfe\x18\xcdY\xf4Pb\xfa\xc3\x8fy%\x00GB-l\x14\xab:\x0f\xeb\xf1\xbb\\/\x10\x8em\x19<W\xc7\xc7+0\xd2.\x06\x9at\x01\xc8Y\xda\xb4Y!n\x0c`>\x88\x14\xea\xf4\x9e\xd4\x90\x8e\xc8\xf3\x92e\x9c+W\xd5\x04\x9d\xf6\xa3\xa7c\xd9\x17W\xf2\x88
}}

The debug log illustrates that AWSCLI is trying to reach out [GetObjectTagging API |https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html] available in AWS S3, which is not available in Apache Ozone.

The solution for this issue would be two-level:
1. Handle unknown get parameters as as unknown in S3Gateway and return some error like 500 or whatever. Pros: easy. Cons: not easy? to list up all 'known' get parameters.
2. Implement tagging API. Pros: right fix. Cons: needs protobuf-format change of Key metadata. Maybe adding some map data structure for user-defined metadata.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org