[GitHub] [arrow-datafusion] alamb commented on a change in pull request #2029: Refactor SessionContext, SessionState and SessionConfig to support multi-tenancy configurations - Part 2

[GitBox <gi...@apache.org>]

alamb commented on a change in pull request #2029:
URL: https://github.com/apache/arrow-datafusion/pull/2029#discussion_r830580391



##########
File path: datafusion/src/execution/context.rs
##########
@@ -846,39 +773,14 @@ pub struct SessionConfig {
     /// parallel using the provided `target_partitions` level
     pub repartition_windows: bool,
     /// Should DataFusion parquet reader using the predicate to prune data
-    parquet_pruning: bool,
-    /// Runtime configurations such as memory threshold and local disk for spill
-    pub runtime: RuntimeConfig,
+    pub parquet_pruning: bool,

Review comment:
       @yjshen I agree that the interface with getters would be good;
   
   ```rust
   pub fn get_bool(&self, key: &str) -> Result<bool> {
       self.get(key).and_then(Value::into_bool)
   }
   ```
   
   Among other things that would allow us to change the implementation (e.g. to properties or typed fields) without issue 
   
   I do think typed fields have some nice properties (like being able to do possibly earlier error detection) but in order to support extensions/customized properties we definitely need to support aribtrary key/values as well
   
   In terms of `TaskProperties` I personally prefer something like this:
   
   ```rust
   pub struct TaskProperties {
       ///SessionConfig
       config: SessionConfig,
       /// Additional Name-value pairs of task properties
       properties: HashMap<String, String>,
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org