You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "dreamtalen (via GitHub)" <gi...@apache.org> on 2025/09/19 22:01:36 UTC
[I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
dreamtalen opened a new issue, #495:
URL: https://github.com/apache/arrow-rs-object-store/issues/495
**Describe the bug**
Multipart upload initialization fails with HTTP 411 (Length Required) error when using Google Cloud Storage (GCS) with S3Store:
```
Generic S3 error: Error performing POST https://storage.googleapis.com/<bucket>/0.binary?uploads= in 117.19262ms - Server returned non-2xx status code: 411 Length Required: <!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 411 (Length Required)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/
googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>411.</b> <ins>That's an error.</ins>
<p>POST requests require a <code>Content-length</code> header. <ins>That's all we know.</ins>
-> Error performing POST https://storage.googleapis.com/<bucket>/0.binary?uploads= in 117.19262ms - Server returned non-2xx status code: 411 Length Required: <!DOCTYPE html>
```
Similar issues have been reported:
- https://github.com/apache/arrow-rs-object-store/issues/278
- https://github.com/delta-io/delta-rs/issues/2394
A fix exists for the GCP client:
- https://github.com/apache/arrow-rs-object-store/pull/257
Would it be possible to apply a similar fix to the AWS client so that GCP can be used in S3-compatible mode with S3Store?
**To Reproduce**
Create a s3 store with AmazonS3Builder and GCP configs:
```
let gcp_s3 = AmazonS3Builder::new()
.with_endpoint("https://storage.googleapis.com")
.with_region(REGION)
.with_bucket_name(BUCKET_NAME)
.with_access_key_id(ACCESS_KEY_ID)
.with_secret_access_key(SECRET_KEY)
.build();
```
Start a Multipart Upload using https://docs.rs/object_store/latest/object_store/#multipart-upload
**Expected behavior**
Multipart upload should work with GCP when used in S3-compatible mode via S3Store.
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
Posted by "dreamtalen (via GitHub)" <gi...@apache.org>.
dreamtalen commented on issue #495:
URL: https://github.com/apache/arrow-rs-object-store/issues/495#issuecomment-3314642585
> Why not just use the GCS native API?
>
> Changing to enhancement as this isn't something we explicitly support.
GoogleCloudStorageBuilder doesn't support GCS HMAC keys (https://cloud.google.com/storage/docs/authentication/hmackeys)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #495: Error 411 (Length Required) when using Multipart PUT on GCP with S3Store
URL: https://github.com/apache/arrow-rs-object-store/issues/495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #495:
URL: https://github.com/apache/arrow-rs-object-store/issues/495#issuecomment-3314638649
Why not just use the GCS native API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
Posted by "dreamtalen (via GitHub)" <gi...@apache.org>.
dreamtalen commented on issue #495:
URL: https://github.com/apache/arrow-rs-object-store/issues/495#issuecomment-3321509197
> Right they're part of the compat API for migrating workloads designed for S3 and which don't support GCS natively, I'm confused as to why you would choose to use such APIs when you don't have to? Using the GCS APIs is better supported, and there is a wider choice of Auth options, many of which avoid the risks of long-lived static credentials such as HMAC keys?
>
> From the provided link
>
> > HMAC keys are useful when you want to move data between other cloud storage providers and Cloud Storage, because HMAC keys allow you to reuse your existing code to access Cloud Storage.
>
> I.e. they're a migration tool
Sure, we're currently in a migration phase to GCS and many of users still rely on S3-based tools like boto3, s5cmd, etc. These tools generally work fine with GCS HMAC keys and we only run into this issue with object_store crate. I have prepared a PR at https://github.com/apache/arrow-rs-object-store/pull/496 to address this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error 411 (Length Required) when using Multipart PUT on GCP with S3Store [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #495:
URL: https://github.com/apache/arrow-rs-object-store/issues/495#issuecomment-3314750991
Right they're part of the compat API for migrating workloads designed for S3 and which don't support GCS natively, I'm confused as to why you would choose to use such APIs when you don't have to? Using the GCS APIs is better supported, and there is a wider choice of Auth options, many of which avoid the risks of long-lived static credentials such as HMAC keys?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org