You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/25 13:45:35 UTC

[GitHub] [arrow-datafusion] andygrove opened a new issue #62: DataFrame.collect() should be extensible

andygrove opened a new issue #62:
URL: https://github.com/apache/arrow-datafusion/issues/62


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   Ballista provides its own execution context but uses the DataFusion DataFrame. Calling `collect` on the DataFrame will run the query in-memory rather than distributed and Ballista users must instead extract the logical plan from the DataFrame and call `BallistaContext.collect` instead. This is not good UX.
   
   **Describe the solution you'd like**
   As a user, I would just like to call `DataFrame.collect()` and have it run either in-memory or distributed depending on how I created the context.
   
   I think the way to do this is by making it possible to customize `ExecutionContext` and override the behavior when a DataFrame is collected.
   
   **Describe alternatives you've considered**
   None
   
   **Additional context**
   None
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] jorgecarleitao commented on issue #62: DataFrame.collect() should be extensible

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on issue #62:
URL: https://github.com/apache/arrow-datafusion/issues/62#issuecomment-826327836


   One way to address this:
   
   * Create a trait `ExecutionContext` with necessary APIs
   * make `DataFrame` hold `Box<dyn ExecutionContext>`
   
   This allows DataFusions' `DataFrame` to have different contexts resolved at run time (e.g. whether `local` or `cluster` is passed?).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove closed issue #62: DataFrame.collect() should be extensible

Posted by GitBox <gi...@apache.org>.
andygrove closed issue #62:
URL: https://github.com/apache/arrow-datafusion/issues/62


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] jorgecarleitao commented on issue #62: DataFrame.collect() should be extensible

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on issue #62:
URL: https://github.com/apache/arrow-datafusion/issues/62#issuecomment-826327836


   One way to address this:
   
   * Create a trait `ExecutionContext` with necessary APIs
   * make `DataFrame` hold `Box<dyn ExecutionContext>`
   
   This allows DataFusions' `DataFrame` to have different contexts resolved at run time (e.g. whether `local` or `cluster` is passed?).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove commented on issue #62: DataFrame.collect() should be extensible

Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #62:
URL: https://github.com/apache/arrow-datafusion/issues/62#issuecomment-912621283


   I am closing this since I found a different solution to this issue. Ballista uses the DataFusion context but provides its own query planner which results in a DistributedQueryExec physical plan, so DataFrame.collect() invokes that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org