You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/14 22:39:33 UTC
[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4633: DataFrame owned SessionState
tustvold commented on code in PR #4633:
URL: https://github.com/apache/arrow-datafusion/pull/4633#discussion_r1049041281
##########
datafusion/core/src/dataframe.rs:
##########
@@ -74,40 +73,22 @@ use crate::prelude::SessionContext;
/// ```
#[derive(Debug, Clone)]
pub struct DataFrame {
- session_state: Arc<RwLock<SessionState>>,
+ session_state: SessionState,
plan: LogicalPlan,
}
impl DataFrame {
/// Create a new Table based on an existing logical plan
- pub fn new(session_state: Arc<RwLock<SessionState>>, plan: LogicalPlan) -> Self {
+ pub fn new(session_state: SessionState, plan: LogicalPlan) -> Self {
Self {
session_state,
plan,
}
}
/// Create a physical plan
- pub async fn create_physical_plan(self) -> Result<Arc<dyn ExecutionPlan>> {
- // this function is copied from SessionContext function of the
- // same name
- let state_cloned = {
- let mut state = self.session_state.write();
- state.execution_props.start_execution();
-
- // We need to clone `state` to release the lock that is not `Send`. We could
- // make the lock `Send` by using `tokio::sync::Mutex`, but that would require to
- // propagate async even to the `LogicalPlan` building methods.
- // Cloning `state` here is fine as we then pass it as immutable `&state`, which
- // means that we avoid write consistency issues as the cloned version will not
- // be written to. As for eventual modifications that would be applied to the
- // original state after it has been cloned, they will not be picked up by the
- // clone but that is okay, as it is equivalent to postponing the state update
- // by keeping the lock until the end of the function scope.
- state.clone()
- };
-
- state_cloned.create_physical_plan(&self.plan).await
+ pub async fn create_physical_plan(&self) -> Result<Arc<dyn ExecutionPlan>> {
+ self.session_state.create_physical_plan(&self.plan).await
Review Comment:
In practice the lock meant that we were still cloning `SessionState` fairly frequently, better to just be explicit about it and optimise from there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org