[GitHub] [arrow-datafusion] yjshen opened a new issue #946: PartitionedFile abstraction for flexible table scan

[GitBox <gi...@apache.org>]

yjshen opened a new issue #946:
URL: https://github.com/apache/arrow-datafusion/issues/946


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   Currently, we can only handle table input on a file basis. A more flexible and extensible abstraction is desirable.
   
   **Describe the solution you'd like**
   ```rust
   pub struct PartitionedFile {
       /// Path for the file (e.g. URL, filesystem path, etc)
       pub file_path: String,
       /// Statistics of the file
       pub statistics: Statistics,
   
       // Several feasible extensions
   
       // 1. partitioned table
       /// Values of partition columns to be appended to each row
       pub partition_value: Option<Vec<ScalarValue>>,
   
       // 2. sub-file processing
       /// A range of file to process in the current task
       pub start: u64,
       pub end: u64,
   }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb closed issue #946: PartitionedFile abstraction for flexible table scan

[GitBox <gi...@apache.org>]

alamb closed issue #946:
URL: https://github.com/apache/arrow-datafusion/issues/946


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org