You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/11/15 04:04:55 UTC

[GitHub] [arrow-datafusion] houqp opened a new issue #1301: Assign stats per partitioned file

houqp opened a new issue #1301:
URL: https://github.com/apache/arrow-datafusion/issues/1301


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   https://github.com/apache/arrow-datafusion/commit/2454e468641d4d98af211c2800c0afec2732385b removed statistics field from `PartitionedFile` struct. It looks like there is no way to associate statis per file in our current design?
   
   **Describe the solution you'd like**
   
   Some table formats like delta lake stores per file stats in its table metadata, this can be used to perform further pruning during table partition scans. Even though we don't support such optimization in Datafusion today, it would be useful to add a public interface to associate stats per partitioned file to lay out the ground work.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] houqp closed issue #1301: Assign stats per partitioned file

Posted by GitBox <gi...@apache.org>.
houqp closed issue #1301:
URL: https://github.com/apache/arrow-datafusion/issues/1301


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] houqp commented on issue #1301: Assign stats per partitioned file

Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1301:
URL: https://github.com/apache/arrow-datafusion/issues/1301#issuecomment-968516923


   @rdettai perhaps I am missing something here, are we managing these per file stats somewhere else now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] houqp commented on issue #1301: Assign stats per partitioned file

Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1301:
URL: https://github.com/apache/arrow-datafusion/issues/1301#issuecomment-976238890


   @rdettai good point, I agree with you that TableProvider is the right place to manage per file stats, and thus file level pruning.  This will be another datafusion in datafusion use-case where the `scan` method will need to write custom code to evaluate the passed in filter expressions against collected file stats.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] rdettai commented on issue #1301: Assign stats per partitioned file

Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #1301:
URL: https://github.com/apache/arrow-datafusion/issues/1301#issuecomment-970049109


   You are correct, we are not managing the per file statistics anywhere. Neither do we have per partition statistics. The only granularity at which we are managing statistics **(for now)** is at the `ExecutionPlan` node level:
   ```Rust
   pub trait ExecutionPlan: Debug + Send + Sync {
       ...
   
       /// Returns the global output statistics for this `ExecutionPlan` node.
       fn statistics(&self) -> Statistics;
   }
   ```
   
   This means that there is no point in providing higher granularity statistics to the `ExecutionPlan`, it wouldn't know what to do with it 😄. The idea is that anything you want to do do with your higher level statistics (pruning,...), you do it at the `TableProvider` level, and then you give the `ExecutionPlan` only the information it needs, that it to say the file statistics aggregated together (you can use `get_statistics_with_limit` to do that).
   
   When (if) we decide that we need higher granularity statistics in the `ExecutionPlan`, we will need to provide the statistics differently. But would that be the file level? I am not sure. It seems to me that the "execution-plan-partition" level will be more appropriate. I guess that the "execution-plan-partition" could be a file, multiple files, or a chunk of a file. This means that statistic would be attached to another struct, that would look like:
   ```Rust
   struct PhysicalPlanFilePartition {
     file_group: Vec<PartitionedFile>,
     statistics: Statistics,
   }
   ```
   
   But again, I really think that this abstraction is worth being introduced only when (if) we decide that we need higher granularity statistics in the `ExecutionPlan`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org