You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2023/10/11 10:26:29 UTC
[I] DynamoDB Lock Support [arrow-rs]
tustvold opened a new issue, #4880:
URL: https://github.com/apache/arrow-rs/issues/4880
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
https://github.com/apache/arrow-rs/pull/4239 added copy_if_not_exists support for S3-compatible stores that support it, i.e. everything other than S3 :sweat_smile:.
Having native support for this, and potentially other conditional operations in the future (#4879), would reduce user friction and I think unlock a number of exciting use-cases by making support consistent across the backends.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I would like to be able to configure a DynamoDB to use in a manner compatible with the [AWS DynamoDB Lock Client](https://github.com/awslabs/amazon-dynamodb-lock-client).
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
These is a [Rust port](https://docs.rs/dynamodb_lock/latest/dynamodb_lock/index.html) of the AWS lock client, but it depends on rusoto, and doesn't appear to have very strong fencing guarantees.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
FYI @wjones127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4880: DynamoDB Lock Support
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4880:
URL: https://github.com/apache/arrow-rs/issues/4880#issuecomment-1741182969
Hmm... Yeah, I spent some time bashing at this, and it isn't easy to provide strong guarantees with this approach :sweat_smile:
As iceberg also is doing something similar using DynamoDB to store the latest metadata file, I think lets not do this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `copy_if_not_exists` support for AmazonS3 via DynamoDB Lock Support [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4880:
URL: https://github.com/apache/arrow-rs/issues/4880#issuecomment-2741683292
Migrating from arrow-rs issue [#4880](https://github.com/apache/arrow-rs/issues/4880)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `copy_if_not_exists` support for AmazonS3 via DynamoDB Lock Support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4880: Add `copy_if_not_exists` support for AmazonS3 via DynamoDB Lock Support
URL: https://github.com/apache/arrow-rs/issues/4880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on issue #4880: DynamoDB Lock Support
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #4880:
URL: https://github.com/apache/arrow-rs/issues/4880#issuecomment-1741173533
FWIW, the DynamoDB lock protocol hasn't become the preferred protocol for Delta Lake or Lance, mostly because of the dangers of lease-based locks. Spark Delta Lake went with a slightly different protocol, [described in this doc](https://docs.google.com/document/d/1Gs4ZsTH19lMxth4BSdwlWjUNR-XhKHicDvBjd2RqNd8/edit#heading=h.mjjuxw9mcz9h). In Lance we did something similar. delta-rs still uses dynamodb-lock, but will eventually move to match the protocol used by Spark Delta Lake.
That being said, offering something decent out-of-the-box for S3 might be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #4880: DynamoDB Lock Support
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4880: DynamoDB Lock Support
URL: https://github.com/apache/arrow-rs/issues/4880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org