[GitHub] [arrow-datafusion] Jimexist commented on a change in pull request #579: WIP use into iter to minimize array concats

[GitBox <gi...@apache.org>]

Jimexist commented on a change in pull request #579:
URL: https://github.com/apache/arrow-datafusion/pull/579#discussion_r656118650



##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -248,21 +252,25 @@ impl AggregateWindowExpr {
         let values = self.evaluate_args(batch)?;
         let results = partition_points
             .iter()
-            .map(|partition_range| {
+            .map::<Result<_>, _>(|partition_range| {
                 let sort_partition_points =
                     find_ranges_in_range(partition_range, &sort_partition_points);
                 let mut window_accumulators = self.create_accumulator()?;
-                sort_partition_points
+                let result = sort_partition_points
                     .iter()
                     .map(|range| window_accumulators.scan_peers(&values, range))
-                    .collect::<Result<Vec<_>>>()
+                    .collect::<Result<Vec<_>>>()?
+                    .into_iter()
+                    .flatten();
+                ScalarValue::iter_to_array(result)

Review comment:
       @Dandandan i find that changing to use this, even if would be saving ArrayRef creating, makes the performance much worse, because it changes from iterating trusted len iterator to this boxed type with unbounded untrusted iterator.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org