You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mpiannucci (via GitHub)" <gi...@apache.org> on 2025/02/17 20:17:20 UTC

[I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

mpiannucci opened a new issue, #7146:
URL: https://github.com/apache/arrow-rs/issues/7146

   Although compatible and available in the options of the object_store google cloud create, there is no way to use etag to conditionally put an object with google cloud storage. 
   
   **Additional context**
   
   Currently the [`put`](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/gcp/client.rs#L378) operation on GoogleCloudStorageClient accepts conditional put via `PutOptions::mode`:
   
   ```rust
           let builder = match &opts.mode {
               PutMode::Overwrite => builder.idempotent(true),
               PutMode::Create => builder.header(&VERSION_MATCH, "0"),
               PutMode::Update(v) => {
                   let etag = v.version.as_ref().ok_or(Error::MissingVersion)?;
                   builder.header(&VERSION_MATCH, etag)
               }
           };
   ```
   
   However, the implementation for `PutMode::Update` accepts an [`UpdateVersion`](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/lib.rs#L1133) but ignores the `e_tag` variant even though it [is supported by google cloud](https://cloud.google.com/storage/docs/request-preconditions#compose-preconditions). 
   
   e-tag should be supported the same as generation match for put updates via google cloud 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

Posted by "mpiannucci (via GitHub)" <gi...@apache.org>.
mpiannucci commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664064005

   So looking at the XML API you are correct https://cloud.google.com/storage/docs/xml-api/put-object-upload
   
   But using the google cloud command line using if-match header manually does indeed work, for example:
   
   ```
   gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt
   # Fails!!!
   gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt --additional-headers=if-match=foo
   # Get etag
   gcloud storage object describe gs://arraylake-external-bucket/test.txt
   # Works!
   gcloud storage cp "test.txt" gs://arraylake-external-bucket/test.txt --additional-headers=if-match=COagt5O8y4sDEAE=
   ```
   
   So not sure if it doesnt work or if its undocumented


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

Posted by "mpiannucci (via GitHub)" <gi...@apache.org>.
mpiannucci commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664083340

   Fair enough for now. Thanks for jumping in here!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #7146: [Object Store] Google Cloud storage doesnt support ETag for conditional put operations
URL: https://github.com/apache/arrow-rs/issues/7146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664069927

   So the official Google Cloud SDKs are documented to not support ETag for mutation, see [here](https://cloud.google.com/python/docs/reference/storage/latest/generation_metageneration#conditional-parameters).
   
   As such this is either some new feature that has been shadow released, or there are subtle edge cases that cause it to not be reliable. In the absence of any documentation, I think we have to assume there is a reason to not use this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Object Store] Google Cloud storage doesnt support ETag for conditional put operations [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7146:
URL: https://github.com/apache/arrow-rs/issues/7146#issuecomment-2664054848

   Google cloud only supports preconditions on generation (i.e. version) not on Etag.
   
   The [docs](https://github.com/apache/arrow-rs/blob/38d6e691f4ee1b356f28d77b6820de67166c51c3/object_store/src/lib.rs#L1130-L1131) on UpdateVersion allude to this
   
   > Stores will use differing combinations of `e_tag` and `version` to provide conditional updates, and it is therefore recommended applications preserve both


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org