You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/29 18:28:41 UTC
[PR] Alamb/issue 18344 opt [datafusion]
alamb opened a new pull request, #18360:
URL: https://github.com/apache/datafusion/pull/18360
## Which issue does this PR close?
- Related to https://github.com/apache/datafusion/issues/18334
- Follow on to https://github.com/apache/datafusion/pull/18356
## Rationale for this change
@mhilton's change to `date_trunc` to make it correct also potentially slows it down
Let's try and recover the performance (and then some) so DataFusion 51 can be both more correct **and** faster
## What changes are included in this PR?
Use `try_unary_op` methods to avoid allocating intermediate arrays to improve performance
## Are these changes tested?
functionally by CI
I will run benchmarks on this PR as well
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3503076894
We went with a dfferent approach , and I have filed the follow on ticket
- https://github.com/apache/arrow-rs/issues/8808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` more [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2474653373
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,56 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
array: &PrimitiveArray<T>,
granularity: &str,
) -> Result<ArrayRef> {
- let unit = match (tu, granularity) {
- (Second, "minute") => Some(Int64Array::new_scalar(60)),
- (Second, "hour") => Some(Int64Array::new_scalar(3600)),
- (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
- (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
- (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
- (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
- (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
- (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
- (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
- (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
- (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
- (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
- (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
- (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
- (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
- (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
- (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
- (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+ let unit: Option<i64> = match (tu, granularity) {
Review Comment:
The key idea here is to make this code faster by reusing the allocation and operating in place rather than allocating new arrays
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3466625831
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/issue-18344-opt (dc2b5f7416940be07b3aa8f4caa09b7b0aaefbc2) to 6cc73fa24d0d4d4006375e3bc20d6c267fc55c44 [diff](https://github.com/apache/datafusion/compare/6cc73fa24d0d4d4006375e3bc20d6c267fc55c44..dc2b5f7416940be07b3aa8f4caa09b7b0aaefbc2)
BENCH_NAME=date_trunc
BENCH_COMMAND=cargo bench --bench date_trunc
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_issue-18344-opt
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2504036359
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
}
}
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+ array: PrimitiveArray<Int64Type>,
+ op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+ F: Fn(i64) -> Result<i64>,
Review Comment:
- I filed https://github.com/apache/arrow-rs/issues/8808 to track this idea
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #18360: Optimize `date_trunc` function by avoiding allocations
URL: https://github.com/apache/datafusion/pull/18360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2475607920
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,57 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
array: &PrimitiveArray<T>,
granularity: &str,
) -> Result<ArrayRef> {
- let unit = match (tu, granularity) {
- (Second, "minute") => Some(Int64Array::new_scalar(60)),
- (Second, "hour") => Some(Int64Array::new_scalar(3600)),
- (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
- (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
- (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
- (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
- (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
- (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
- (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
- (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
- (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
- (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
- (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
- (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
- (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
- (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
- (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
- (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+ let unit: Option<i64> = match (tu, granularity) {
+ (Second, "minute") => Some(60),
+ (Second, "hour") => Some(3600),
+ (Second, "day") => Some(86400),
+
+ (Millisecond, "second") => Some(1_000),
+ (Millisecond, "minute") => Some(60_000),
+ (Millisecond, "hour") => Some(3_600_000),
+ (Millisecond, "day") => Some(86_400_000),
+
+ (Microsecond, "millisecond") => Some(1_000),
+ (Microsecond, "second") => Some(1_000_000),
+ (Microsecond, "minute") => Some(60_000_000),
+ (Microsecond, "hour") => Some(3_600_000_000),
+ (Microsecond, "day") => Some(86_400_000_000),
+
+ (Nanosecond, "microsecond") => Some(1_000),
+ (Nanosecond, "millisecond") => Some(1_000_000),
+ (Nanosecond, "second") => Some(1_000_000_000),
+ (Nanosecond, "minute") => Some(60_000_000_000),
+ (Nanosecond, "hour") => Some(3_600_000_000_000),
+ (Nanosecond, "day") => Some(86_400_000_000_000),
_ => None,
};
if let Some(unit) = unit {
let original_type = array.data_type();
- let array = arrow::compute::cast(array, &DataType::Int64)?;
- let array = arrow::compute::kernels::numeric::div(&array, &unit)?;
- let array = arrow::compute::kernels::numeric::mul(&array, &unit)?;
+ let input = arrow::compute::cast(array, &DataType::Int64)?;
+ // Optimize performance by doing operations in place if possible
+ let array = input.as_primitive::<Int64Type>().clone();
+ drop(input); // ensure the input reference is dropped (so we can reuse the memory if possible)
+ let array = try_unary_mut_or_clone(array, |i| {
+ i.checked_div(unit)
+ .ok_or_else(|| exec_datafusion_err!("division overflow"))
+ })?;
+ let array = try_unary_mut_or_clone(array, |i| {
Review Comment:
This is true, though I don't know how to represent this in code.
Maybe I could make a second function `try_unary_mut_or_error` that throws a runtime error 🤔
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
}
}
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+ array: PrimitiveArray<Int64Type>,
+ op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+ F: Fn(i64) -> Result<i64>,
Review Comment:
yes, I agree -- the try_unary_mut is quite awkward to use. I will see if I can port some of these changes upstream / see what they look like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#issuecomment-3466642250
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group alamb_issue-18344-opt main
----- --------------------- ----
date_trunc_minute_1000 1.10 5.2±0.02µs ? ?/sec 1.00 4.8±0.01µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Optimize `date_trunc` function by avoiding allocations [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #18360:
URL: https://github.com/apache/datafusion/pull/18360#discussion_r2475557450
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -493,6 +515,21 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
}
}
+/// Applies the unary operation in place if possible, or cloning the array if not
+fn try_unary_mut_or_clone<F>(
+ array: PrimitiveArray<Int64Type>,
+ op: F,
+) -> Result<PrimitiveArray<Int64Type>>
+where
+ F: Fn(i64) -> Result<i64>,
Review Comment:
not really date_trunc specific. can this be made more flexible with a more generous use of generics?
perhaps it could even be in arrow-rs. it makes `try_unary_mut` significantly more approachable
##########
datafusion/functions/src/datetime/date_trunc.rs:
##########
@@ -455,36 +456,57 @@ fn general_date_trunc_array_fine_granularity<T: ArrowTimestampType>(
array: &PrimitiveArray<T>,
granularity: &str,
) -> Result<ArrayRef> {
- let unit = match (tu, granularity) {
- (Second, "minute") => Some(Int64Array::new_scalar(60)),
- (Second, "hour") => Some(Int64Array::new_scalar(3600)),
- (Second, "day") => Some(Int64Array::new_scalar(86400)),
-
- (Millisecond, "second") => Some(Int64Array::new_scalar(1_000)),
- (Millisecond, "minute") => Some(Int64Array::new_scalar(60_000)),
- (Millisecond, "hour") => Some(Int64Array::new_scalar(3_600_000)),
- (Millisecond, "day") => Some(Int64Array::new_scalar(86_400_000)),
-
- (Microsecond, "millisecond") => Some(Int64Array::new_scalar(1_000)),
- (Microsecond, "second") => Some(Int64Array::new_scalar(1_000_000)),
- (Microsecond, "minute") => Some(Int64Array::new_scalar(60_000_000)),
- (Microsecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000)),
- (Microsecond, "day") => Some(Int64Array::new_scalar(86_400_000_000)),
-
- (Nanosecond, "microsecond") => Some(Int64Array::new_scalar(1_000)),
- (Nanosecond, "millisecond") => Some(Int64Array::new_scalar(1_000_000)),
- (Nanosecond, "second") => Some(Int64Array::new_scalar(1_000_000_000)),
- (Nanosecond, "minute") => Some(Int64Array::new_scalar(60_000_000_000)),
- (Nanosecond, "hour") => Some(Int64Array::new_scalar(3_600_000_000_000)),
- (Nanosecond, "day") => Some(Int64Array::new_scalar(86_400_000_000_000)),
+ let unit: Option<i64> = match (tu, granularity) {
+ (Second, "minute") => Some(60),
+ (Second, "hour") => Some(3600),
+ (Second, "day") => Some(86400),
+
+ (Millisecond, "second") => Some(1_000),
+ (Millisecond, "minute") => Some(60_000),
+ (Millisecond, "hour") => Some(3_600_000),
+ (Millisecond, "day") => Some(86_400_000),
+
+ (Microsecond, "millisecond") => Some(1_000),
+ (Microsecond, "second") => Some(1_000_000),
+ (Microsecond, "minute") => Some(60_000_000),
+ (Microsecond, "hour") => Some(3_600_000_000),
+ (Microsecond, "day") => Some(86_400_000_000),
+
+ (Nanosecond, "microsecond") => Some(1_000),
+ (Nanosecond, "millisecond") => Some(1_000_000),
+ (Nanosecond, "second") => Some(1_000_000_000),
+ (Nanosecond, "minute") => Some(60_000_000_000),
+ (Nanosecond, "hour") => Some(3_600_000_000_000),
+ (Nanosecond, "day") => Some(86_400_000_000_000),
_ => None,
};
if let Some(unit) = unit {
let original_type = array.data_type();
- let array = arrow::compute::cast(array, &DataType::Int64)?;
- let array = arrow::compute::kernels::numeric::div(&array, &unit)?;
- let array = arrow::compute::kernels::numeric::mul(&array, &unit)?;
+ let input = arrow::compute::cast(array, &DataType::Int64)?;
+ // Optimize performance by doing operations in place if possible
+ let array = input.as_primitive::<Int64Type>().clone();
+ drop(input); // ensure the input reference is dropped (so we can reuse the memory if possible)
+ let array = try_unary_mut_or_clone(array, |i| {
+ i.checked_div(unit)
+ .ok_or_else(|| exec_datafusion_err!("division overflow"))
+ })?;
+ let array = try_unary_mut_or_clone(array, |i| {
Review Comment:
technically speaking, only the first `try_unary_mut_or_clone` is needed
on the second transformation, we're guaranteed to be the pointer into the array, and the so taking the `or_clone` path would be an error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org