You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "criccomini (via GitHub)" <gi...@apache.org> on 2025/05/03 17:54:30 UTC

[I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

criccomini opened a new issue, #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347

   **Which part is this question about**
   <!--
   Is it code base, library api, documentation or some other part?
   -->
   
   ObjectStore's parse_url.
   
   **Describe your question**
   <!--
   A clear and concise description of what the question is.
   -->
   
   We're looking at converting between `ObjectStore`s and URLs in SlateDB:
   
   https://github.com/slatedb/slatedb/pull/533#issuecomment-2848730594
   
   URL to `ObjectStore` seems pretty straight forward with `parse_url`, but it's  unclear how we might go from `ObjectStore` to URL. Is there some `get_url()` method we can call on the `ObjectStore` trait? I poked around but didn't find anything. I suspect I'm missing something obvious.
   
   **Additional context**
   <!--
   Add any other context about the problem here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #347: Is there a way to go from `ObjectStore` to `(URL, opts)`?
URL: https://github.com/apache/arrow-rs-object-store/issues/347


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "criccomini (via GitHub)" <gi...@apache.org>.
criccomini commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2851756054

   Oh, thanks for the pointer. `ObjectStoreRegistry` looks exactly like what we'd want. Upstreaming it would be really nice. Would you be open to a PR? (I don't want to deal with updating the data fusion code, though 😅 )
   
   Regarding the comment in #259, I'm playing around with the code a bit just to get a feel for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "criccomini (via GitHub)" <gi...@apache.org>.
criccomini commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2851678587

   Gotcha. Would you be open to a PR to generate/expose the URL via a `get_url` method in the `ObjectStore` trait? The reason I ask is that SlateDB takes `ObjectStore`s in its constructor. Users are constructing the ObjectStore outside of our code. We could, of course, ask them to supply a url and opts, and call `parse_url` while caching the URL. But that seems less desirable since it makes it harder to do dependency injection for testing. It also feels a bit clunky.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2849055301

   This is not currently supported, you will need to store the URL alongside the ObjectStire in whatever you are using to cache them.
   
   See also https://github.com/apache/arrow-rs-object-store/issues/259


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2855228298

   > Oh, thanks for the pointer. `ObjectStoreRegistry` looks exactly like what we'd want. Upstreaming it would be really nice. Would you be open to a PR? (I don't want to deal with updating the data fusion code, though 😅 )
   > 
   > Regarding the comment in [#259](https://github.com/apache/arrow-rs-object-store/issues/259), I'm playing around with the code a bit just to get a feel for it.
   
   Just adding the ObjectStoreRegistry to this create is enough -- we can find relevant DataFusion community members to update DataFusion
   
   Thank you for your help @criccomini 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2851894520

   If someone wanted to upstream it, I think that would be valuable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2855234844

   For anyone else curious, here is a link to the docs for [`ObjectStoreRegistry`](https://docs.rs/datafusion/47.0.0/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2851685437

   See https://github.com/apache/arrow-rs-object-store/issues/259#issuecomment-2741746698
   
   TLDR I am not sure how this would actually work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Is there a way to go from `ObjectStore` to `(URL, opts)`? [arrow-rs-object-store]

Posted by "criccomini (via GitHub)" <gi...@apache.org>.
criccomini commented on issue #347:
URL: https://github.com/apache/arrow-rs-object-store/issues/347#issuecomment-2855498435

   PR here: https://github.com/apache/arrow-rs-object-store/pull/348


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org