You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/21 16:59:14 UTC
[GitHub] [arrow-datafusion] tustvold opened a new issue #1636: Provide RuntimeEnv to ExecutionContext
tustvold opened a new issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
https://github.com/apache/arrow-datafusion/pull/1526 added a MemoryManager along with the concept of a RuntimeEnv, this is awesome. Unfortunately `ExecutionContext::new()` and `ExecutionContext::with_config` call `RuntimeEnv::new()`, which in turn creates a new `MemoryManager` and `DiskManager`, which create a temporary directory and print logs.
At least within IOx it is common to construct an `ExecutionContext` for each query, as various settings may change, most notably the catalog. Therefore these logs and especially the file IO are prohibitive
**Describe the solution you'd like**
I think it should be possible to pass an extant RuntimeEnv to an ExecutionContext, so that multiple different `ExecutionContext` can share the same resource pool.
**Describe alternatives you've considered**
We could maintain separate ExecutionContext for each type of query IOx executes, but this is just punting the problem imo. We want to be able to have queries running in different contexts, but on the same pool of resources managed by a `RuntimeEnv`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1020402603
PR with proposed improvements: https://github.com/apache/arrow-datafusion/pull/1668#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1018744736
Being able to do something like
```rust
let context = context.with_runtime(existing_runtime);
```
makes a lot of sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1020402603
PR with proposed improvements: https://github.com/apache/arrow-datafusion/pull/1668#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1019397093
Thanks for the detailed explanation of the use case in Influx, haven't thought of it before. I think it would be great to have this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb closed issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1020202023
In general I am planning a cleanup to the `MemoryManager` and `DiskManager` configuration. I expect to have a PR for review shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1636: Provide RuntimeEnv to ExecutionContext
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1636:
URL: https://github.com/apache/arrow-datafusion/issues/1636#issuecomment-1020148505
I plan to work on this issue, with respect to https://github.com/influxdata/influxdb_iox/issues/3507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org