[GitHub] [arrow-datafusion] wjones127 commented on issue #2185: ObjectStore write support

[GitBox <gi...@apache.org>]

wjones127 commented on issue #2185:
URL: https://github.com/apache/arrow-datafusion/issues/2185#issuecomment-1094096751

   > I wonder what quality of write support do you plan to provide ?
   
   Basically, I would like for `ObjectStore` to be the Rust Datafusion equivalent of [Arrow C++'s FileSystem](https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/filesystem.h) or Python's [fsspec](https://filesystem-spec.readthedocs.io/en/latest/). They provide a common interface to various object stores (S3, GCS, ADLS, HDFS, etc.) so that various projects implementing readers and writers (such as delta-rs) can simply use those filesystems instead of taking on the burden of writing and maintaining all those abstractions themselves.
   
   > Production ready implementation of data ingestion can be as large effort as having to create another project like Apache Kafka.
   
   This is just the "filesystem" interaction, so just reading and writing bytes to various places with a uniform API. Other "writer" related things like file formats (parquet / json / csv) would be out of scope. Does that make sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org