You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/17 08:36:09 UTC

[GitHub] [arrow-datafusion] rdettai opened a new issue #1012: Make the conversion from logical to physical plan async

rdettai opened a new issue #1012:
URL: https://github.com/apache/arrow-datafusion/issues/1012


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   As discussed in #1010, it would add a lot of flexibility to the planning to make it possible to use async methods. Thanks to #962, the statistics resolution can be delayed to the creation of the physical plan. Adding async to `TableProvider.scan()`, the method that converts the logical plan to the physical plan, will make it possible to have async statistics resolution. 
   
   **Describe the solution you'd like**
   Make the trait method `TableProvider.scan()` async and propagate that change to the outer APIs (mostly `ExecutionContext.create_physical_plan(&LogicalPlan)`)
   
   **Describe alternatives you've considered**
   - also add statistics to optimizers: that could be a next step but it is not required for now
   - leave the planning sync and force `TableProvider` implementers to make their async methods sync: syncifying in Rust is pretty hard ([cloud-reader-rs](https://github.com/cloudfuse-io/cloud-readers-rs) is an example of this) and it is a huge constraint for implementers 
   
   **Additional context**
   This will also help the useage of the `ObjectStore` abstraction added in #811.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb closed issue #1012: Make the conversion from logical to physical plan async

Posted by GitBox <gi...@apache.org>.
alamb closed issue #1012:
URL: https://github.com/apache/arrow-datafusion/issues/1012


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] rdettai edited a comment on issue #1012: Make the conversion from logical to physical plan async

Posted by GitBox <gi...@apache.org>.
rdettai edited a comment on issue #1012:
URL: https://github.com/apache/arrow-datafusion/issues/1012#issuecomment-921820012


   This also pops a question. The state of `ExecutionContext` is protected by a Mutex:
   ```Rust
   pub struct ExecutionContext {
       /// Internal state for the context
       pub state: Arc<Mutex<ExecutionContextState>>,
   }
   ```
   
   Why do some of its methods take `&mut self` ???


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] rdettai commented on issue #1012: Make the conversion from logical to physical plan async

Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #1012:
URL: https://github.com/apache/arrow-datafusion/issues/1012#issuecomment-921820012


   This also pops a question. The state of execution context is protected by a Mutex:
   ```Rust
   pub struct ExecutionContext {
       /// Internal state for the context
       pub state: Arc<Mutex<ExecutionContextState>>,
   }
   ```
   
   Why do some of its methods take `&mut self` ???


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] houqp commented on issue #1012: Make the conversion from logical to physical plan async

Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1012:
URL: https://github.com/apache/arrow-datafusion/issues/1012#issuecomment-923166578


   > Why do some of its methods take &mut self ???
   
   Probably for no good reason, maybe it used to not be protected by mutex.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org