You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2025/01/20 02:13:07 UTC
[I] [C++] Compatibility with newer aws sdk [arrow]
h-vetinari opened a new issue, #45304:
URL: https://github.com/apache/arrow/issues/45304
### Describe the enhancement requested
Not sure whether to classify this as a bug or as an enhancement. It feels like a regression, but likely in AWS itself. In conda-forge, we don't build all patch versions of the SDK, because it gets released way too often to migrate ever time. Recently we went from 1.1.458 to 1.1.488, which ran into consistent errors on unix.
Failures look roughly like this
```
[----------] 28 tests from TestS3FS
[ RUN ] TestS3FS.GetFileInfoRoot
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
[ FAILED ] TestS3FS.GetFileInfoRoot (43 ms)
[ RUN ] TestS3FS.GetFileInfoBucket
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
[ FAILED ] TestS3FS.GetFileInfoBucket (52 ms)
```
all due to `malformed chunked encoding`
This also shows up in the python tests (these are from arrow v15, which is the last version where we built the python bindings on the same feedstock; since the errors are uniform across versions though, I strongly suspect this to be the case also for main, as well as v19 and down).
```
=========================== short test summary info ============================
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[cloudpickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_dataset.py::test_write_dataset_s3 - OSError: When creating key 'dataset/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_create_dir[S3FileSystem] - OSError: When creating key 'test-directory/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_copy_file[S3FileSystem] - OSError: When uploading part for key 'test-copy-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_move_file[S3FileSystem] - OSError: When uploading part for key 'test-move-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_delete_file[S3FileSystem] - OSError: When uploading part for key 'test-delete-target-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_from_uri_s3 - OSError: When creating key 'foo/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
```
Xref also #45195
### Component(s)
C++, Python, Packaging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601917573
I have it down to this HTTP exchange:
* Request by AWS SDK:
```
PUT /bucket/emptydir/ HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: B545DDA1-9E33-4987-B074-2B147AB8BD75
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=5be6a95be231511f69fdc161843bbe83413ba52540e1607bc9de36ea42a827c5
content-encoding: aws-chunked
content-type: binary/octet-stream
transfer-encoding: chunked
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date: 20250120T094325Z
x-amz-decoded-content-length: 0
x-amz-trailer: x-amz-checksum-crc64nvme
Expect: 100-continue
```
* Response by Minio:
```
HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 331
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF688A7B0
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT
Connection: close
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BadRequest</Code><Message>malformed chunked encoding</Message><Key>emptydir/</Key><BucketName>bucket</BucketName><Resource>/bucket/emptydir/</Resource><RequestId>181C5D5BF688A7B0</RequestId><HostId>a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf</HostId></Error>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601892849
Ok, I can reproduce, and even with the latest Minio version is still occurs. So I'll probably have to run Wireshark on this...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2602071796
> Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.
Thanks for digging into this so quickly! Could you open an issue in https://github.com/aws/aws-sdk-cpp? That would be useful for tracking/fixing this - if you want I can also do it, but I'd only be able to do a pale approximation, given that you understand the subject much better and did all the debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2602047212
Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2607743569
Issue resolved by pull request 45310
https://github.com/apache/arrow/pull/45310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2602089973
I'll open an issue, yes. I think I've found the culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "xhochy (via GitHub)" <gi...@apache.org>.
xhochy commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601928226
Would it help your debugging if I build some of the versions in between on conda-forge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601957979
Probably not, thanks for the proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601678070
The error message [comes from Minio](https://github.com/minio/minio/blob/779ec8f0d423ee8114ef752628f4208c69c78445/cmd/streaming-signature-v4.go#L183-L184). So this is probably an incompatibility between Minio and the official AWS SDK... :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2602239150
Reported upstream at https://github.com/aws/aws-sdk-cpp/issues/3259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2602004503
It appears there might be an easy workaround for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45304:
URL: https://github.com/apache/arrow/issues/45304#issuecomment-2601924216
Interesting, another PUT request before (to create a bucket) is successful.
* Request:
```http
PUT /empty-bucket HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: 54812C34-9549-4ECB-B54E-15B98759BCB9
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=65a56d02027d24a6b29815814d051b012501bdb545bd32dc051f2f4c7b01ee7a
content-length: 0
content-type: application/xml
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-api-version: 2006-03-01
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20250120T094325Z
```
* Response:
```http
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Location: /empty-bucket
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF637BDF2
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Compatibility with newer aws sdk [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #45304: [C++] Compatibility with newer aws sdk
URL: https://github.com/apache/arrow/issues/45304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org