You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "ZetaTwo (via GitHub)" <gi...@apache.org> on 2023/05/01 20:51:44 UTC

[GitHub] [arrow-rs] ZetaTwo opened a new issue, #4162: Ability to get actually full path/URL, e.g. from ObjectMeta

ZetaTwo opened a new issue, #4162:
URL: https://github.com/apache/arrow-rs/issues/4162

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   It would be nice to have a function/attribute to give me the full URL for an object. Currently, the ObjectMeta.location provides the path _within_ the object store but I would like to get the full URL such as "gs://bucket/dir/file.txt".
   
   **Describe the solution you'd like**
   
   Maybe add an attribute "uri" to the ObjectMeta struct or provide a new function `.get_uri` on the ObjectStore struct.
   
   **Additional context**
   
   The reason I am looking for this is that I need to pass the URL a different program and it would be nice to keep all the logic centralised in the object_store library instead of having a separate function with "if gcp: "gs://" + path elif aws: "s3://" + path" etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #4162: Ability to get actually full path/URL, e.g. from ObjectMeta

Posted by "tustvold (via GitHub)" <gi...@apache.org>.

tustvold commented on issue #4162:
URL: https://github.com/apache/arrow-rs/issues/4162#issuecomment-1531432581

   I think we definitely should better flesh out the URL handling story in object_store, #4047 is also in a similar vein.
   
   There are a couple of things though that make this challenging:
   
   * The URLs with custom schemes are not especially well standardised, for example, `abfs://` has two completely different conventions
   * Store adapters like PrefixStore can't meaningfully reason about how to add a prefix to a URL returned by the inner store
   * The stores themselves are not URL-aware - see #3784
   
   I wonder if you've given thought to simply storing the base URL alongside the `ObjectStore`, to allow constructing the URL in the form expected by the different program? Just spit-balling here...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] ZetaTwo commented on issue #4162: Ability to get actually full path/URL, e.g. from ObjectMeta

Posted by "ZetaTwo (via GitHub)" <gi...@apache.org>.

ZetaTwo commented on issue #4162:
URL: https://github.com/apache/arrow-rs/issues/4162#issuecomment-1532039710

   Yeah I understand that there might be some complications in implementing this for all possible backends. I can totally store the URL in a different place but I just thought it would have been cool to get the info directly from the backend.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Ability to get actually full path/URL, e.g. from ObjectMeta [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.

tustvold commented on issue #4162:
URL: https://github.com/apache/arrow-rs/issues/4162#issuecomment-1788637034

   A further wrinkle here, is that some tools such as the AWS CLI are actually not following the URL specification, and are interpreting the URL paths verbatim.
   
   See https://github.com/apache/arrow-datafusion/issues/8009 and https://github.com/apache/arrow-rs/issues/5017.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org