You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mpiannucci (via GitHub)" <gi...@apache.org> on 2025/02/17 20:17:20 UTC
[I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
mpiannucci opened a new issue, #7146:
URL: https://github.com/apache/arrow-rs/issues/7146
Although compatible and available in the options of the object_store google cloud create, there is no way to use etag to conditionally put an object with google cloud storage.
**Additional context**
Currently the [`put`](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/gcp/client.rs#L378) operation on GoogleCloudStorageClient accepts conditional put via `PutOptions::mode`:
```rust
let builder = match &opts.mode {
PutMode::Overwrite => builder.idempotent(true),
PutMode::Create => builder.header(&VERSION_MATCH, "0"),
PutMode::Update(v) => {
let etag = v.version.as_ref().ok_or(Error::MissingVersion)?;
builder.header(&VERSION_MATCH, etag)
}
};
```
However, the implementation for `PutMode::Update` accepts an [`UpdateVersion`](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/lib.rs#L1133) but ignores the `e_tag` variant even though it [is supported by google cloud](https://cloud.google.com/storage/docs/request-preconditions#compose-preconditions).
e-tag should be supported the same as generation match for put updates via google cloud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
Posted by "mpiannucci (via GitHub)" <gi...@apache.org>.
mpiannucci commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664064005
So looking at the XML API you are correct https://cloud.google.com/storage/docs/xml-api/put-object-upload
But using the google cloud command line using if-match header manually does indeed work, for example:
```
gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt
# Fails!!!
gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt --additional-headers=if-match=foo
# Get etag
gcloud storage object describe gs://arraylake-external-bucket/test.txt
# Works!
gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt --additional-headers=if-match=COagt5O8y4sDEAE=
```
So not sure if it doesnt work or if its undocumented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
Posted by "mpiannucci (via GitHub)" <gi...@apache.org>.
mpiannucci commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664083340
Fair enough for now. Thanks for jumping in here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #7146: [Object Store] Google Cloud storage doesnt support ETag for conditional put operations
URL: https://github.com/apache/arrow-rs/issues/7146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664069927
So the official Google Cloud SDKs are documented to not support ETag for mutation, see [here](https://cloud.google.com/python/docs/reference/storage/latest/generation_metageneration#conditional-parameters).
As such this is either some new feature that has been shadow released, or there are subtle edge cases that cause it to not be reliable. In the absence of any documentation, I think we have to assume there is a reason to not use this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664054848
Google cloud only supports preconditions on generation (i.e. version) not on Etag.
The [docs](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/lib.rs#L1130-L1131) on UpdateVersion allude to this
> Stores will use differing combinations of `e_tag` and `version` to provide conditional updates, and it is therefore recommended applications preserve both
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org