You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/25 21:43:09 UTC

[GitHub] [arrow-datafusion] alamb opened a new pull request #1680: Use NamedTempFile rather than `String` in DiskManager

alamb opened a new pull request #1680:
URL: https://github.com/apache/arrow-datafusion/pull/1680


   # Which issue does this PR close?
   
   Closes https://github.com/apache/arrow-datafusion/issues/1679
   
    # Rationale for this change
   1. Using `String` for temporary files leaves the files around longer than necessary, and would cause trouble with a long lived DiskManager across plans.
   2. Using the existing `tempfile` crate in rust is likely to work better across operating systems than DataFusion specific tempfile creation logic
   
   # What changes are included in this PR?
   1. `DiskManager` passes out `NamedTempFile`s rather than `String` (when these are dropped they also clean up the temp file immediately)
   2. Update users to use NamedTempFiles
   
   # Are there any user-facing changes?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on pull request #1680: Use NamedTempFile rather than `String` in DiskManager

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1680:
URL: https://github.com/apache/arrow-datafusion/pull/1680#issuecomment-1021636850


   cc @yjshen 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1680: Use NamedTempFile rather than `String` in DiskManager

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1680:
URL: https://github.com/apache/arrow-datafusion/pull/1680#discussion_r792144273



##########
File path: datafusion/src/execution/disk_manager.rs
##########
@@ -120,34 +116,15 @@ fn create_local_dirs(local_dirs: Vec<PathBuf>) -> Result<Vec<TempDir>> {
         .collect()
 }
 
-fn get_file(file_name: &str, local_dirs: &[TempDir]) -> String {
-    let mut hasher = DefaultHasher::new();
-    file_name.hash(&mut hasher);
-    let hash = hasher.finish();
-    let dir = &local_dirs[hash.rem_euclid(local_dirs.len() as u64) as usize];
-    let mut path = PathBuf::new();
-    path.push(dir);
-    path.push(file_name);
-    path.to_str().unwrap().to_string()
-}
+fn create_tmp_file(local_dirs: &[TempDir]) -> Result<NamedTempFile> {

Review comment:
       tempfiles are now generated using `tempfile` rather than string manipulation




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb merged pull request #1680: Use NamedTempFile rather than `String` in DiskManager

Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1680:
URL: https://github.com/apache/arrow-datafusion/pull/1680


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1680: Use NamedTempFile rather than `String` in DiskManager

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1680:
URL: https://github.com/apache/arrow-datafusion/pull/1680#discussion_r792144774



##########
File path: datafusion/src/physical_plan/sorts/sort.rs
##########
@@ -301,17 +306,16 @@ async fn spill_partial_sorted_stream(
 }
 
 fn read_spill_as_stream(
-    path: String,
+    path: NamedTempFile,

Review comment:
       Since ownership of `NamedTempFile` is passed into the actual task doing the reading, so when it is done, the temp file is cleaned up 🧹 

##########
File path: datafusion/src/physical_plan/sorts/sort.rs
##########
@@ -301,17 +306,16 @@ async fn spill_partial_sorted_stream(
 }
 
 fn read_spill_as_stream(
-    path: String,
+    path: NamedTempFile,

Review comment:
       Since ownership of `NamedTempFile` is passed into the actual task doing the reading, when it is done, the temp file is cleaned up 🧹 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org