[GitHub] [arrow-datafusion] korowa commented on a diff in pull request #5937: Streaming Memory Reservation in SHJ

["korowa (via GitHub)" <gi...@apache.org>]

korowa commented on code in PR #5937:
URL: https://github.com/apache/arrow-datafusion/pull/5937#discussion_r1163829779


##########
datafusion/core/src/physical_plan/joins/symmetric_hash_join.rs:
##########
@@ -1442,6 +1478,9 @@ impl SymmetricHashJoinStream {
                     // Combine results:
                     let result =
                         combine_two_batches(&self.schema, equal_result, anti_result)?;
+                    let capacity = self.size();
+                    self.metrics.stream_memory_usage.set(capacity);
+                    self.reservation.lock().resize(capacity);

Review Comment:
   Wouldn't it be more appropriate to use fallible version of reserving memory function (like `try_grow`) instead of for cases of increasing reservation size? I guess, sharing memory pool across multiple operators, will still fail execution, but `try_grow` usage seems to look more precise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org