[GitHub] [arrow-ballista] milenkovicm commented on issue #6: [Ballista] Support to access remote object store, like HDFS, S3, etc

[GitBox <gi...@apache.org>]

milenkovicm commented on issue #6:
URL: https://github.com/apache/arrow-ballista/issues/6#issuecomment-1234425565

   Had a quick look into this issue, and from what I can see, there is not nothing missing on datafusion side to have this functionality (apart from some hard work :)).
   
   Team did a great job to add support for object store in datafusion:
   
   ```rust
   use std::sync::Arc;
   use datafusion::{
       datasource::listing::{ListingTable, ListingTableConfig, ListingTableUrl},
       prelude::SessionContext,
   };
   use log::info;
   use object_store::aws::AmazonS3Builder;
   
       let ctx = SessionContext::new();
   
       let s3 = AmazonS3Builder::new()
           .with_region("us-east-1")
           .with_bucket_name("testbucket")
           .with_access_key_id("MINIO")
           .with_secret_access_key("MINIO/MINIO")
           .with_endpoint("http://localhost:9000")
           .with_allow_http(true)
           .build()
           .unwrap();
   
       let s3 = Arc::new(s3);
   
       ctx.runtime_env()
           .register_object_store("s3", "localhost:9000", s3);
   
       let url = ListingTableUrl::parse("s3://localhost:9000/testpath/").unwrap();
   
       let config = ListingTableConfig::new(url)
           .infer(&ctx.state())
           .await
           .unwrap();
   
       let table = ListingTable::try_new(config).unwrap();
       ctx.register_table("test", Arc::new(table)).unwrap();
   
       ctx.sql("SELECT * FROM test")
           .await
           .unwrap()
           .show()
           .await
           .unwrap();
   ```
   
   I give quick try with ballista `standalone`, changing code a bit to expose `RuntimeEnv` on client, scheduler, and executor and registering store on each of them manually. At the end, it did produce correct result. Currently getting to a `RuntimeEnv` is not "walk in a park", few hacks here and there were needed, but it is not hard to make it easier. It would then be possible load object store providers from configuration files.
   
   Alternatively `register_object_store` can be provided directly on the `BallistaContext` and then somehow object store configuration may be magically serialized and handled on other actors in the system. `AmazonS3Builder` should probably be modified so it can be serialized.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org