You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "nkemnitz (via GitHub)" <gi...@apache.org> on 2026/06/25 16:04:01 UTC
[PR] fix: accept GCS gzip responses without Content-Length [arrow-rs-object-store]
nkemnitz opened a new pull request, #782:
URL: https://github.com/apache/arrow-rs-object-store/pull/782
# Which issue does this PR close?
Part of #774 (does not fully close it — see below).
# Rationale for this change
GCS serves large objects stored with `Content-Encoding: gzip` using chunked
transfer with no `Content-Length` (and decompressive transcoding when the client
does not accept gzip encoding). `ObjectStore::get`/`head` on GCS required
`Content-Length` unconditionally and failed with
`Generic { store: "GCS", source: Header { source: MissingContentLength } }`,
even though a chunked, self-delimiting body is a valid response (RFC 9112 §6.2
forbids `Content-Length` alongside `Transfer-Encoding: chunked`).
# What changes are included in this PR?
`HeaderConfig` gains `stored_size_header: Option<&'static str>`. When
`Content-Length` is absent, `header_meta` reads the object size from this header.
GCS sets it to `x-goog-stored-content-length` (always present); S3, Azure and the
HTTP store leave it `None`, so a missing `Content-Length` stays a hard error for
them.
# Are there any user-facing changes?
`get()`/`head()` now succeed on chunked gzip GCS objects. On a server-decompressed
(transcoded) read, `ObjectMeta.size` is the *stored* (compressed) size, since the
decompressed length is not known without reading the body; on a passthrough read
(`Accept-Encoding: gzip`) it is exact.
**Not fully resolved:** some transcoded GCS responses (default reads without
`Accept-Encoding: gzip`) also omit the ETag entirely and still fail with
`MissingEtag`. Left for a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: accept GCS gzip responses without Content-Length [arrow-rs-object-store]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #782:
URL: https://github.com/apache/arrow-rs-object-store/pull/782#discussion_r3559043790
##########
src/client/header.rs:
##########
@@ -139,8 +147,12 @@ pub(crate) fn header_meta(
Err(e) => return Err(e),
};
+ // Prefer `Content-Length`, falling back to a store-provided size header: GCS omits
Review Comment:
is there documentation on this behavior you can link to so we can verify this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: accept GCS gzip responses without Content-Length [arrow-rs-object-store]
Posted by "nkemnitz (via GitHub)" <gi...@apache.org>.
nkemnitz commented on code in PR #782:
URL: https://github.com/apache/arrow-rs-object-store/pull/782#discussion_r3641517697
##########
src/client/header.rs:
##########
@@ -139,8 +147,12 @@ pub(crate) fn header_meta(
Err(e) => return Err(e),
};
+ // Prefer `Content-Length`, falling back to a store-provided size header: GCS omits
Review Comment:
The description for what `x-goog-stored-content-length` contains: https://docs.cloud.google.com/storage/docs/xml-api/reference-headers#xgoogstoredcontentlength
In the Google Cloud Rust repo, a code maintainer reports that this header is indeed always present: https://github.com/googleapis/google-cloud-rust/discussions/3713#discussioncomment-14883469
The part that confirms that `Content-Length` and `Content-Encoding` get dropped for decompressive transcoding: https://docs.cloud.google.com/storage/docs/transcoding#decompressive_transcoding
(Although I don't really care about decompressive transcoding - it creates large transfers and as described under the same link in the paragraph below, it prevents integrity checks)
I could not find any official documentation regarding the chunked Transfer-Encoding switch... only that it does... Could weaken that claim to "Google _may_ omit the `Content-Length`".
Other finds:
* [google-cloud-cpp](https://github.com/googleapis/google-cloud-cpp/blob/95511adfa1a6e733ca20dd17aaefd898611fe2f5/google/cloud/storage/internal/rest/object_read_source.cc#L47-L68) prefers `x-goog-stored-content-length` > `content-range` > `content-length`
* [google-cloud-rust](https://github.com/googleapis/google-cloud-rust/blob/18cd4258a67b424a814d29c4b63fad501952c43d/src/storage/src/storage/read_object/parse_http_response.rs#L42) only uses x-goog-stored-content-length and nothing else?
* Their [python code](https://github.com/googleapis/python-storage/blob/ab4997ce0f7b85947e84b226bd0edf6d714a946a/google/cloud/storage/_media/requests/download.py#L163) seems to use it to distinguish truncated downloads from transcoded ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: accept GCS gzip responses without Content-Length [arrow-rs-object-store]
Posted by "nkemnitz (via GitHub)" <gi...@apache.org>.
nkemnitz commented on code in PR #782:
URL: https://github.com/apache/arrow-rs-object-store/pull/782#discussion_r3641517697
##########
src/client/header.rs:
##########
@@ -139,8 +147,12 @@ pub(crate) fn header_meta(
Err(e) => return Err(e),
};
+ // Prefer `Content-Length`, falling back to a store-provided size header: GCS omits
Review Comment:
The description for what `x-goog-stored-content-length` contains: https://docs.cloud.google.com/storage/docs/xml-api/reference-headers#xgoogstoredcontentlength
In the Google Cloud Rust repo, a code maintainer reports that this header is indeed always present: https://github.com/googleapis/google-cloud-rust/discussions/3713#discussioncomment-14883469
The part that confirms that `Content-Length` and `Content-Encoding` get dropped for decompressive transcoding: https://docs.cloud.google.com/storage/docs/transcoding#decompressive_transcoding
(Although I don't really care about decompressive transcoding - it creates large transfers and as described under the same link in the paragraph below, it prevents integrity checks)
I could not find any official documentation regarding the chunked Transfer-Encoding switch... only can see that it does... Could weaken that claim to "Google _may_ omit the `Content-Length`".
Other finds:
* [google-cloud-cpp](https://github.com/googleapis/google-cloud-cpp/blob/95511adfa1a6e733ca20dd17aaefd898611fe2f5/google/cloud/storage/internal/rest/object_read_source.cc#L47-L68) prefers `x-goog-stored-content-length` > `content-range` > `content-length`
* [google-cloud-rust](https://github.com/googleapis/google-cloud-rust/blob/18cd4258a67b424a814d29c4b63fad501952c43d/src/storage/src/storage/read_object/parse_http_response.rs#L42) only uses x-goog-stored-content-length and nothing else?
* Their [python code](https://github.com/googleapis/python-storage/blob/ab4997ce0f7b85947e84b226bd0edf6d714a946a/google/cloud/storage/_media/requests/download.py#L163) seems to use it to distinguish truncated downloads from transcoded ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org