[GitHub] [arrow-datafusion] kmitchener commented on pull request #3540: feat: allow object store registration from datafusion-cli

[GitBox <gi...@apache.org>]

kmitchener commented on PR #3540:
URL: https://github.com/apache/arrow-datafusion/pull/3540#issuecomment-1252211707

   I think that rather than any options, the support for object_store should be invisible to the user. So just running this SQL from datafusion-cli should be enough to trigger the cli to register the s3 objectstore:
   
   ```
   create external table unicorns stored as parquet location 's3://my_bucket/lineitem/';
   ```
   
   You could do that by creating the logical plan from the user's given SQL, then only registering an object store if they're creating an external location that's in the cloud. Something like:
   
   ```rust
       let logical_plan = ctx.create_logical_plan(sql)?;
       match logical_plan {
           LogicalPlan::CreateExternalTable(external) => {
               if external.location.to_lowercase().starts_with("s3://") {
                   // use https://docs.rs/object_store/latest/object_store/path/struct.Path.html
                   // parse the path, find the bucket
                   // register the object store 
               }
           }
           _ => {},
       }
       // continue on and execute the logical plan
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org