[GitHub] [arrow-datafusion] alamb commented on issue #907: S3 Support

[GitBox <gi...@apache.org>]

alamb commented on issue #907:
URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902214790


   > Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions
   
   Yes I think separate crates is a very good idea to avoid datafusion requiring a massive dependency stack and to keep compile time reasonable. 
   
   > How would you think of keeping the code as part of the workspace?
   
   I think this is a reasonable for "core" integrations, though going this path also subjects those crates to the apache arrow governance model (among other things slower release cycle) which may not be needed here
   
   Maybe we could start it as its own repo / separate crate with whoever does it and then we can  figure out if we want to bring it back into apache arrow if any.
   
   > was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this.
   
   FWIW we took this approach in IOx (starting with Rusto) and it has been working well for us: https://github.com/influxdata/influxdb_iox/blob/main/object_store/Cargo.toml#L21-L23 (we have been using it in AWS for several months now and we have not hit any issues with it)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org