You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/29 18:28:41 UTC

[PR] Alamb/issue 18344 opt [datafusion]

alamb opened a new pull request, #18360:
URL: https://github.com/apache/datafusion/pull/18360

   ## Which issue does this PR close?
   
   - Related to https://github.com/apache/datafusion/issues/18334
   - Follow on to https://github.com/apache/datafusion/pull/18356
   
   ## Rationale for this change
   @mhilton's change to `date_trunc` to make it correct also potentially slows it down
   
   Let's try and recover the performance (and then some) so DataFusion 51 can be both more correct **and** faster
   
   ## What changes are included in this PR?
   
   Use `try_unary_op` methods to avoid allocating intermediate arrays to improve performance
   
   ## Are these changes tested?
   
   functionally by CI
   
   I will run benchmarks on this PR as well
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3503076894

   We went with a dfferent approach , and I have filed the follow on ticket
   - https://github.com/apache/arrow-rs/issues/8808


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` more [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2474653373


##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,56 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     array: &PrimitiveArray<T>,
     granularity: &str,
 ) -> Result<ArrayRef> {
-    let unit = match (tu, granularity) {
-        (Second, "minute") => Some(Int64Array::new_scalar(60)),
-        (Second, "hour") => Some(Int64Array::new_scalar(3600)),
-        (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
-        (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
-        (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
-        (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
-        (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
-        (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
-        (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
-        (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
-        (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
-        (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
-        (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
-        (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
-        (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
-        (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
-        (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
-        (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+    let unit: Option<i64> = match (tu, granularity) {

Review Comment:
   The key idea here is to make this code faster by reusing the allocation and operating in place rather than allocating new arrays 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3466625831

   🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
   Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing alamb/issue-18344-opt (dc2b5f7416940be07b3aa8f4caa09b7b0aaefbc2) to 6cc73fa24d0d4d4006375e3bc20d6c267fc55c44 [diff](https://github.com/apache/datafusion/compare/6cc73fa24d0d4d4006375e3bc20d6c267fc55c44..dc2b5f7416940be07b3aa8f4caa09b7b0aaefbc2)
   BENCH_NAME=date_trunc
   BENCH_COMMAND=cargo bench --bench date_trunc
   BENCH_FILTER=
   BENCH_BRANCH_NAME=alamb_issue-18344-opt
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2504036359


##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     }
 }
 
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+    array: PrimitiveArray<Int64Type>,
+    op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+    F: Fn(i64) -> Result<i64>,

Review Comment:
   - I filed https://github.com/apache/arrow-rs/issues/8808 to track this idea
   - 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #18360:  Optimize `date_trunc` function by avoiding allocations
URL: https://github.com/apache/datafusion/pull/18360


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2475607920


##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,57 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     array: &PrimitiveArray<T>,
     granularity: &str,
 ) -> Result<ArrayRef> {
-    let unit = match (tu, granularity) {
-        (Second, "minute") => Some(Int64Array::new_scalar(60)),
-        (Second, "hour") => Some(Int64Array::new_scalar(3600)),
-        (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
-        (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
-        (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
-        (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
-        (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
-        (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
-        (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
-        (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
-        (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
-        (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
-        (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
-        (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
-        (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
-        (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
-        (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
-        (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+    let unit: Option<i64> = match (tu, granularity) {
+        (Second, "minute") => Some(60),
+        (Second, "hour") => Some(3600),
+        (Second, "day") => Some(86400),
+
+        (Millisecond, "second") => Some(1_000),
+        (Millisecond, "minute") => Some(60_000),
+        (Millisecond, "hour") => Some(3_600_000),
+        (Millisecond, "day") => Some(86_400_000),
+
+        (Microsecond, "millisecond") => Some(1_000),
+        (Microsecond, "second") => Some(1_000_000),
+        (Microsecond, "minute") => Some(60_000_000),
+        (Microsecond, "hour") => Some(3_600_000_000),
+        (Microsecond, "day") => Some(86_400_000_000),
+
+        (Nanosecond, "microsecond") => Some(1_000),
+        (Nanosecond, "millisecond") => Some(1_000_000),
+        (Nanosecond, "second") => Some(1_000_000_000),
+        (Nanosecond, "minute") => Some(60_000_000_000),
+        (Nanosecond, "hour") => Some(3_600_000_000_000),
+        (Nanosecond, "day") => Some(86_400_000_000_000),
         _ => None,
     };
 
     if let Some(unit) = unit {
         let original_type = array.data_type();
-        let array = arrow::compute::cast(array, &DataType::Int64)?;
-        let array = arrow::compute::kernels::numeric::div(&array, &unit)?;
-        let array = arrow::compute::kernels::numeric::mul(&array, &unit)?;
+        let input = arrow::compute::cast(array, &DataType::Int64)?;
+        // Optimize performance by doing operations in place if possible
+        let array = input.as_primitive::<Int64Type>().clone();
+        drop(input); // ensure the input reference is dropped (so we can reuse the memory if possible)
+        let array = try_unary_mut_or_clone(array, |i| {
+            i.checked_div(unit)
+                .ok_or_else(|| exec_datafusion_err!("division overflow"))
+        })?;
+        let array = try_unary_mut_or_clone(array, |i| {

Review Comment:
   This is true, though I don't know how to represent this in code. 
   
   Maybe I could make a second function `try_unary_mut_or_error` that throws a runtime error 🤔 



##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     }
 }
 
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+    array: PrimitiveArray<Int64Type>,
+    op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+    F: Fn(i64) -> Result<i64>,

Review Comment:
   yes, I agree -- the  try_unary_mut is quite awkward to use. I will see if I can port some of these changes upstream / see what they look like



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3466642250

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                     alamb_issue-18344-opt                  main
   -----                     ---------------------                  ----
   date_trunc_minute_1000    1.10      5.2±0.02µs        ? ?/sec    1.00      4.8±0.01µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2475557450


##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     }
 }
 
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+    array: PrimitiveArray<Int64Type>,
+    op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+    F: Fn(i64) -> Result<i64>,

Review Comment:
   not really date_trunc specific. can this be made more flexible with a more generous use of generics?
   perhaps it could even be in arrow-rs. it makes `try_unary_mut` significantly more approachable



##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,57 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
     array: &PrimitiveArray<T>,
     granularity: &str,
 ) -> Result<ArrayRef> {
-    let unit = match (tu, granularity) {
-        (Second, "minute") => Some(Int64Array::new_scalar(60)),
-        (Second, "hour") => Some(Int64Array::new_scalar(3600)),
-        (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
-        (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
-        (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
-        (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
-        (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
-        (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
-        (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
-        (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
-        (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
-        (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
-        (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
-        (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
-        (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
-        (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
-        (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
-        (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+    let unit: Option<i64> = match (tu, granularity) {
+        (Second, "minute") => Some(60),
+        (Second, "hour") => Some(3600),
+        (Second, "day") => Some(86400),
+
+        (Millisecond, "second") => Some(1_000),
+        (Millisecond, "minute") => Some(60_000),
+        (Millisecond, "hour") => Some(3_600_000),
+        (Millisecond, "day") => Some(86_400_000),
+
+        (Microsecond, "millisecond") => Some(1_000),
+        (Microsecond, "second") => Some(1_000_000),
+        (Microsecond, "minute") => Some(60_000_000),
+        (Microsecond, "hour") => Some(3_600_000_000),
+        (Microsecond, "day") => Some(86_400_000_000),
+
+        (Nanosecond, "microsecond") => Some(1_000),
+        (Nanosecond, "millisecond") => Some(1_000_000),
+        (Nanosecond, "second") => Some(1_000_000_000),
+        (Nanosecond, "minute") => Some(60_000_000_000),
+        (Nanosecond, "hour") => Some(3_600_000_000_000),
+        (Nanosecond, "day") => Some(86_400_000_000_000),
         _ => None,
     };
 
     if let Some(unit) = unit {
         let original_type = array.data_type();
-        let array = arrow::compute::cast(array, &DataType::Int64)?;
-        let array = arrow::compute::kernels::numeric::div(&array, &unit)?;
-        let array = arrow::compute::kernels::numeric::mul(&array, &unit)?;
+        let input = arrow::compute::cast(array, &DataType::Int64)?;
+        // Optimize performance by doing operations in place if possible
+        let array = input.as_primitive::<Int64Type>().clone();
+        drop(input); // ensure the input reference is dropped (so we can reuse the memory if possible)
+        let array = try_unary_mut_or_clone(array, |i| {
+            i.checked_div(unit)
+                .ok_or_else(|| exec_datafusion_err!("division overflow"))
+        })?;
+        let array = try_unary_mut_or_clone(array, |i| {

Review Comment:
   technically speaking, only the first `try_unary_mut_or_clone` is needed
   on the second transformation, we're guaranteed to be the pointer into the array, and the so taking the `or_clone` path would be an error



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org