You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/19 14:30:24 UTC
[GitHub] [arrow-datafusion] ehenry2 opened a new issue #907: S3 Support
ehenry2 opened a new issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907
Now that there is a PR in-flight for remote storage systems (#811 ), I wanted to ask what the plan was to implement s3 support. Is this something that will live within the datafusion project or maintained outside? Also, does anyone plan on working on this, or is this something that you are looking for a contributor for?
Also, I'm new to the project, so what is the best way to ask these kind of questions...github issues or on the email list? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ehenry2 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
ehenry2 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-919574441
@alamb That makes sense, my thought though is that s3 support can be worked on in parallel (although not able to utilized until the table providers are updated) as the scaffolding is there with the Object Reader and Object Store traits. In order to do that, however, either a repo would need to be created (or one of the other approaches given previously adopted).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000922002
@matthewmturner we created https://github.com/datafusion-contrib to host these community maintained extensions so it's easier for us to evolve the core datafusion code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000937286
> I also recommend taking a look at the new official AWS rust SDK, it seems a lot more mature since we started this discussion.
Ok. Will check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] seddonm1 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005340751
Hi guys,
I built this which works correctly and uses the offical Amazon SDK: https://gist.github.com/seddonm1/2fb5a6892989fe7bf246022a7bd586ee
If it is useful I'm happy to donate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902473395
> Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at ballista-compute/sqlparser-rs#308 (comment)? We could use it as the incubator for various experiments that need collaborations instead of using a personal namespace.
Wow, that's great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1004945831
@houqp @alamb im very interested in getting s3 / glue support added to datafusion cli. do you know of a way to feature gate other libraries without adding their code to datafusion cli codebase? im just concerned that datafusion cli codebase could become messy if we end up adding the code and feature gate for each functionality like this (i.e. aws s3, aws glue, azure blob, custom sql, etc...). or maybe thats okay? what do you think?
As an aside, and for your information, someone emailed me and wanted to add azure blob storage support to datafusion but said they could work on s3 first. ive created a repo in the `datafusion-contrib` organization (https://github.com/datafusion-contrib/datafusion-objectstore-s3) and will point them to the Influxdb IOx implementation for inspiration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner edited a comment on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000961002
@houqp Separate but related to s3 I'm interested in adding the ability to register a AWS glue catalog / database.
Is that type of functionality something I could expect datafusion to have? If so, do you think it makes sense to bundle with s3 functionality? I would of course start with s3. But just for planning purposes (I.e looking at rusoto vs AWS sdk)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001107516
My suggestion would be naming the repos by extension types, for example: `datafusion-objectstore-s3`, `datafusion-table-glue`, etc. But I don't have a strong opinion on this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] nitisht commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
nitisht commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1030531868
> you could also look at our tests to see all functionality
Thank you for this pointer @matthewmturner . This is precisely what I was looking for, everything working great in my local testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ehenry2 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
ehenry2 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-919433781
Wanted to reopen the discussion here...do we have a consensus on which approach we want to go with? Now that the first set of remote storage PRs are merged, this can now be implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000885940
@alamb @houqp I'm really interested in getting s3 support added.
If no one else is working on this I should be able to pick it up soon (of course I'll likely have questions along the way). Based on the above it seems the approach would be to add it as crate in arrow-datafusion. Assuming this is still the case is there anything else I should know before starting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] seddonm1 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005405304
@houqp thanks have joined that org.
@matthewmturner I have updated my example with the tests I ran against the Minio docker container (https://docs.min.io/docs/minio-docker-quickstart-guide) (so ignore the passwords as they are just the suggested values). I suggest something similar for testing in CI.
I can do a PR tomorrow AU time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] seddonm1 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1006127388
for those waiting for this functionality a PR has been raised: https://github.com/datafusion-contrib/datafusion-objectstore-s3/pull/2
If you want to help test you can follow the instructions in the PR and it should work. The abstraction is good @rdettai and will be even better once the `async fn chunk_reader` can be used 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp closed issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp closed issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] nitisht commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
nitisht commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1029832007
Folks, thank you for great work on this issue. I see that repo https://github.com/datafusion-contrib/datafusion-objectstore-s3 has the relevant code added. I am a little hazy on how do I get these two projects (datafusion & datafusion-objectstore-s3) compiled and work together. Are there any pointers towards this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902085957
> My personal feeling is that we probably don't want to have every storage system inside the DataFusion crate by default but either:
>
>
>
> * Keep separate crates, like `datafusion-s3` which has implementations for some interfaces, and maybe also exposes some helper functions
>
> * Have some optional features (e.g. `s3`) living inside the crate.
>
>
>
> Wydt @alamb @houqp @andygrove @jorgecarleitao
A separate crate is a good idea.
How would you think of keeping the code as part of the workspace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902018820
My personal feeling is that we probably don't want to have every storage system inside the DataFusion crate by default but either:
* Keep separate crates, like `datafusion-s3` which has implementations for some interfaces, and maybe also exposes some helper functions
* Have some optional features (e.g. `s3`) living inside the crate.
Wydt @alamb @houqp @andygrove @jorgecarleitao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902694324
> Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at [sqlparser-rs/sqlparser-rs#308 (comment)](https://github.com/sqlparser-rs/sqlparser-rs/issues/308#issue-868292203)? We could use it as the incubator for various experiments that need collaborations instead of using personal namespace.
If we are developing crates that are extensions to DataFusion then I would be nervous about creating a separate org & community with its own governance for these crates. I think it would be fine for generic database-related crates that don't have "arrow" or "datafusion" in the name, such as the `sqlparser-rs` crate.
We already have the ability to create `arrow-experimental-*` repos where we do not need so much governance. Perhaps we should try that route first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ehenry2 commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
ehenry2 commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902093313
Another question I wanted to throw out too, on implementation details (probably putting the cart before the horse, but wanted to ask anyway), was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000922083
I also recommend taking a look at the new official AWS rust SDK, it seems a lot more mature since we started this discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005429069
@seddonm1 ok i will look into that. also once im comfortable with the usage i can work on the documentation.
from a code perspective, is there anything in particular that we should be aware of? i.e. any issues with aws rust sdk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005192362
FYI there is some great work from @yahoNanJing in this area on https://github.com/apache/arrow-datafusion/pull/1062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1030096196
@nitisht the README on https://github.com/datafusion-contrib/datafusion-objectstore-s3 has some examples how to use it and you could also look at our tests to see all functionality that we have tested (https://github.com/datafusion-contrib/datafusion-objectstore-s3/blob/main/src/object_store/aws.rs).
If there is something that is not clear - or a functionality you need that we havent yet added could you please create an issue in that repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001208143
> FWIW one example of using the aws s3 object store interface is in IOx: https://github.com/influxdata/influxdb_iox/blob/main/object_store/src/aws.rs in case that is helpful
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001111141
makes sense. do you think for glue it might be better to use catalog instead of table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-919484240
I think there is a consensus on the architecture within datafusion, now that https://github.com/apache/arrow-datafusion/pull/950 has been merged
I think we now would need to rework the various table providers (`CsvExec`, `ParquetExec`, etc) to use that interface rather than `File` or `Read` directly, and then we would be ready to create an S3 plugin for DataFusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] rdettai commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-926512141
#1010 (more exactly https://github.com/rdettai/arrow-datafusion/pull/1) is taking care of re-organizing the `TableProvider`s and `ExecutionPlan`s to integrate the `ObjectStore` abstraction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1030097989
@alamb @houqp im thinking we can close this issue now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005368607
@seddonm1 I invited you to the datafusion contrib org as well, could you work with @matthewmturner to figure out how you want to collaborate on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp edited a comment on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp edited a comment on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000976441
I think so, but i think the GLUE catalog integration should be implemented as a table provider plugin in a separate crate, not as part of the s3 object store plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902214790
> Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions
Yes I think separate crates is a very good idea to avoid datafusion requiring a massive dependency stack and to keep compile time reasonable.
> How would you think of keeping the code as part of the workspace?
I think this is a reasonable for "core" integrations, though going this path also subjects those crates to the apache arrow governance model (among other things slower release cycle) which may not be needed here
Maybe we could start it as its own repo / separate crate with whoever does it and then we can figure out if we want to bring it back into apache arrow if any.
> was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this.
FWIW we took this approach in IOx (starting with Rusto) and it has been working well for us: https://github.com/influxdata/influxdb_iox/blob/main/object_store/Cargo.toml#L21-L23 (we have been using it in AWS for several months now and we have not hit any issues with it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902468450
I also think separate crate is a good idea. The design that @yjshen came up with made it very easy to implement different remote storage backends as self-contained plugins.
Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at https://github.com/ballista-compute/sqlparser-rs/issues/308#issue-868292203? We could use it as the incubator for various experiments that need collaborations instead of using personal namespace.
> How would you think of keeping the code as part of the workspace?
I think the upside of this is we won't need to deal with IP clearance if we ever need to donate it back to datafusion. The downside is what Andrew said earlier, slower release cycle due to apache governance overhead, which is not too bad for a stable project. For newer and fast evolving projects, it might slow things down unnecessarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-919885653
Starting to work on an experimental crate within `arrow-datafusion` seems like a good approach to me.
I haven't tracked / studied the requirements for releasing alpha/betas in fine detail, but I think as long as they are clearly marked as not official apache releases, it would be fine to put them on crates.io
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-919727206
Taking @andygrove 's comment into account, unless anyone volunteers to help create the `arrow-experimental-datafusion-s3` repo, it might be easier to follow @Jimexist 's suggestion to create and manage the s3 plugin crate inside datafusion's workspace. Downstream applications can pull in the plugin dependency by git hash if we are behind on official releases. Also I believe we can release alpha/beta versions to crates.io without requiring an official vote?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] gopik commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
gopik commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005407820
Thanks @seddonm1. minio provides s3 api compatibility for azure blob storage. I'll try running the tests today with azure blob storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000961002
@houqp Separate but related to s3 I'm interested in adding the ability to register a AWS glue catalog / database.
Is that type of functionality something I could expect datafusion to have? If so, do you think it makes sense to bundle with s3? I would of course start with s3. But just for planning purposes (I.e looking at rusoto vs AWS sdk)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001112002
Yeah, i think that's a better name :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902517921
If someone may be interested, I will also work on HDFS support right after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001101092
@houqp for these cloud based services what do you think about a naming convention like the following:
datafusion-[cloud provider]-[service]
i.e.
datafusion-aws-s3
datafusion-aws-glue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1000976441
I think so, but i think the GLUE catalog integration should be implemented as a table provider plugin in a separate crate, not object store plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1001180637
FWIW one example of using the aws s3 object store interface is in IOx: https://github.com/influxdata/influxdb_iox/blob/main/object_store/src/aws.rs in case that is helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] matthewmturner commented on issue #907: S3 Support
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-1005374151
@houqp yup sounds good.
@seddonm1 that would be fantastic if you could donate it. do you have a preference for how you would like to do that? if you want to contribute directly i could work on setting up CI in the repo and testing what you've done on my side.
let me know how you would like to proceed and thanks so much for the donation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org