You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2026/10/23 21:32:49 UTC
[PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
mbutrovich opened a new pull request, #26686:
URL: https://github.com/apache/datafusion/pull/26686
## Which issue does this PR close?
- Closes #26678.
- Closes #14738.
- Closes #26685.
## Rationale for this change
These fixes depend on each other. #26678 and #14738 come from the same [NULL-as-zero branch](https://github.com/apache/datafusion/blob/dad9063becc0b10550936425d8b433046c5622e3/datafusion/functions/src/datetime/date_part.rs#L490-L499). Fixing #14738 makes `millisecond` and `microsecond` of a long duration return NULL instead of a wrapped value. Without the #26685 fix, that NULL fails with `declared as non-nullable` on a non-nullable input, so fixing #14738 alone would replace a wrong result with an error.
## What changes are included in this PR?
- `millisecond`, `microsecond` and `nanosecond` share one function that reuses arrow's `date_part` and `cast` where they compute the right value.
- A value that overflows the return type is NULL, as arrow already does for durations. Should overflow be an error instead? That would change other parts too, so I'd rather do it separately.
- The integer parts of `date_part` and the Spark `date_part` are nullable again, as before #19823 (53.0.0). `epoch` keeps the input's nullability.
- The first commit (c920a5b92) adds benchmark cases without the fix, so the benchmark bot can compare against it.
## What is the testing strategy for this PR?
New cases in `datafusion/sqllogictest/test_files/datetime/date_part.slt` and a unit test in `datafusion/spark/src/function/datetime/date_part.rs`.
## Are there any user-facing changes?
`EXTRACT(MILLISECOND/MICROSECOND/NANOSECOND)` on `Duration` and `Interval` return correct results, and long durations return NULL instead of a wrapped value or a panic. The result of `date_part` (except `epoch`) and of the Spark `date_part` is nullable, as before 53.0.0. There are no public API changes.
## AI Disclosure
I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375404753
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375349435)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6375349435-3809-phgtw 6.12.105+ #1 SMP Sat Sep 5 07:54:11 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384714775
Thanks for the review @comphead!
> The doc on `subsecond_part` still explains Int64 by the 60-second bound.
Updated in [`date_part.rs`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/functions/src/datetime/date_part.rs#L517-L520).
> Coverage gaps worth one query each: `nanosecond` on a `MonthDayNano` past minus one minute (only -1.5 s is tested, and the `% 60_000_000_000` arm is new), `nanosecond` on a negative `DayTime` and one with a non-zero days field, `Timestamp(Microsecond)` sub-second parts, and `Time32`/`Time64` sub-second parts on an array with a NULL row.
Added [negative and multi-day `DayTime` and `MonthDayNano` cases](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1327-L1353), [one query per timestamp unit with a NULL row](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1029-L1083), and [one query per time unit with a NULL row](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L915-L950). The pre-1970 and Monrovia queries now cast a column per unit, because `VALUES` had coerced every row to `Timestamp(ns)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384399183
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384373752)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6384373752-3831-7nxvj 6.12.105+ #1 SMP Sat Sep 5 07:57:24 UTC 2026 x86_64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384373752
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
resources:
arch: amd64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375741870
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375352980)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.05 354.6±1.31ns ? ?/sec 1.00 336.3±1.03ns ? ?/sec
date_part_microsecond_date64_1000 1.08 359.5±1.60ns ? ?/sec 1.00 333.5±1.08ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 5.38 5.5±0.01µs ? ?/sec 1.00 1020.3±2.71ns ? ?/sec
date_part_microsecond_interval_dt_1000 9.17 7.5±0.09µs ? ?/sec 1.00 815.6±3.04ns ? ?/sec
date_part_microsecond_interval_mdn_1000 4.70 8.3±0.09µs ? ?/sec 1.00 1766.3±3.75ns ? ?/sec
date_part_microsecond_ms_1000 1.17 23.6±0.02µs ? ?/sec 1.00 20.3±0.02µs ? ?/sec
date_part_microsecond_ns_1000 1.17 21.9±0.01µs ? ?/sec 1.00 18.7±0.01µs ? ?/sec
date_part_microsecond_ns_tz_1000 1.06 93.8±0.26µs ? ?/sec 1.00 88.4±0.26µs ? ?/sec
date_part_microsecond_s_1000 1.29 10.5±0.01µs ? ?/sec 1.00 8.2±0.01µs ? ?/sec
date_part_microsecond_time32_ms_1000 2.34 6.0±0.00µs ? ?/sec 1.00 2.6±0.00µs ? ?/sec
date_part_microsecond_time64_ns_1000 2.24 6.0±0.01µs ? ?/sec 1.00 2.7±0.00µs ? ?/sec
date_part_microsecond_us_1000 1.18 22.4±0.11µs ? ?/sec 1.00 19.1±0.01µs ? ?/sec
date_part_millisecond_date32_1000 1.04 352.5±1.20ns ? ?/sec 1.00 339.2±1.52ns ? ?/sec
date_part_millisecond_date64_1000 1.11 353.7±1.29ns ? ?/sec 1.00 317.5±0.64ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 4.62 4.7±0.01µs ? ?/sec 1.00 1012.6±2.71ns ? ?/sec
date_part_millisecond_interval_dt_1000 9.61 7.4±0.09µs ? ?/sec 1.00 774.9±7.59ns ? ?/sec
date_part_millisecond_interval_mdn_1000 4.69 8.3±0.10µs ? ?/sec 1.00 1770.4±3.52ns ? ?/sec
date_part_millisecond_ms_1000 1.11 22.5±0.02µs ? ?/sec 1.00 20.4±0.17µs ? ?/sec
date_part_millisecond_ns_1000 1.12 21.1±0.05µs ? ?/sec 1.00 18.9±0.02µs ? ?/sec
date_part_millisecond_ns_tz_1000 1.06 94.0±0.35µs ? ?/sec 1.00 88.3±0.15µs ? ?/sec
date_part_millisecond_s_1000 1.27 10.5±0.01µs ? ?/sec 1.00 8.3±0.01µs ? ?/sec
date_part_millisecond_time32_ms_1000 2.11 5.2±0.01µs ? ?/sec 1.00 2.5±0.00µs ? ?/sec
date_part_millisecond_time64_ns_1000 1.95 5.2±0.01µs ? ?/sec 1.00 2.6±0.00µs ? ?/sec
date_part_millisecond_us_1000 1.13 21.6±0.02µs ? ?/sec 1.00 19.1±0.02µs ? ?/sec
date_part_nanosecond_date32_1000 1.06 384.1±1.36ns ? ?/sec 1.00 363.1±0.96ns ? ?/sec
date_part_nanosecond_date64_1000 1.07 386.6±1.21ns ? ?/sec 1.00 360.8±0.82ns ? ?/sec
date_part_nanosecond_duration_ns_1000 29.06 6.5±0.06µs ? ?/sec 1.00 224.7±0.65ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 32.41 8.4±0.06µs ? ?/sec 1.00 259.5±1.06ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 9.41 2.1±0.00µs ? ?/sec 1.00 227.5±0.83ns ? ?/sec
date_part_nanosecond_interval_dt_1000 9.60 7.0±0.06µs ? ?/sec 1.00 731.4±0.93ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 6.48 7.9±0.07µs ? ?/sec 1.00 1219.2±0.94ns ? ?/sec
date_part_nanosecond_ms_1000 1.00 20.2±0.03µs ? ?/sec 1.00 20.2±0.02µs ? ?/sec
date_part_nanosecond_ns_1000 1.00 18.5±0.01µs ? ?/sec 1.00 18.4±0.02µs ? ?/sec
date_part_nanosecond_ns_tz_1000 1.04 91.7±0.36µs ? ?/sec 1.00 88.2±0.38µs ? ?/sec
date_part_nanosecond_s_1000 1.00 8.3±0.01µs ? ?/sec 1.00 8.3±0.03µs ? ?/sec
date_part_nanosecond_time32_ms_1000 1.01 2.6±0.00µs ? ?/sec 1.00 2.5±0.00µs ? ?/sec
date_part_nanosecond_time64_ns_1000 1.02 2.6±0.00µs ? ?/sec 1.00 2.5±0.00µs ? ?/sec
date_part_nanosecond_us_1000 1.00 19.1±0.03µs ? ?/sec 1.00 19.0±0.02µs ? ?/sec
date_part_second_date32_1000 1.02 358.6±2.06ns ? ?/sec 1.00 351.2±1.35ns ? ?/sec
date_part_second_date64_1000 1.00 10.3±0.05µs ? ?/sec 1.01 10.4±0.03µs ? ?/sec
date_part_second_ms_1000 1.00 10.3±0.02µs ? ?/sec 1.00 10.2±0.03µs ? ?/sec
date_part_second_ns_1000 1.00 9.6±0.01µs ? ?/sec 1.00 9.6±0.01µs ? ?/sec
date_part_second_s_1000 1.01 7.8±0.01µs ? ?/sec 1.00 7.7±0.02µs ? ?/sec
date_part_second_us_1000 1.00 9.7±0.02µs ? ?/sec 1.01 9.8±0.01µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 675.1s |
| Peak memory | 35.4 MiB |
| Avg memory | 20.8 MiB |
| CPU user | 546.9s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 675.2s |
| Peak memory | 35.6 MiB |
| Avg memory | 21.5 MiB |
| CPU user | 549.7s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367247937
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
Review Comment:
> These two arms can be one:
Done in [`date_part.rs`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/functions/src/datetime/date_part.rs#L540-L544). It costs about 300 ns per batch on `nanosecond` of `Duration(ns)`, from 227 ns in the [earlier run](https://github.com/apache/datafusion/pull/26686#issuecomment-6375740311) to 542 ns in the [latest](https://github.com/apache/datafusion/pull/26686#issuecomment-6384580919) on Neoverse-V2 at 1000 rows. The cost is fixed per batch, so I kept the simpler form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367251070
##########
datafusion/spark/src/function/datetime/date_part.rs:
##########
@@ -81,10 +81,10 @@ impl ScalarUDFImpl for SparkDatePart {
internal_err!("Use return_field_from_args in this case instead.")
}
- fn return_field_from_args(&self, args: ReturnFieldArgs) -> Result<FieldRef> {
- let nullable = args.arg_fields.iter().any(|f| f.is_nullable());
-
- Ok(Arc::new(Field::new(self.name(), DataType::Int32, nullable)))
+ fn return_field_from_args(&self, _args: ReturnFieldArgs) -> Result<FieldRef> {
+ // Nullable like the integer parts of `date_part`, which this simplifies
+ // to, because they can be NULL for a non-null input
+ Ok(Arc::new(Field::new(self.name(), DataType::Int32, true)))
}
Review Comment:
> So `return_type` could return `Ok(DataType::Int32)` and the override could go.
Done in [`date_part.rs`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/spark/src/function/datetime/date_part.rs#L79-L84). A [`DESCRIBE` query](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/spark/datetime/date_part.slt#L288-L293) replaces the unit test.
> Another option is to delegate to `datafusion_functions::datetime::date_part().return_field_from_args(args)`, which keeps the two matching by construction.
The `nanosecond` and `epoch` mismatch is also on `main` ([Spark](https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/spark/src/function/datetime/date_part.rs#L84-L87), [`date_part`](https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/functions/src/datetime/date_part.rs#L194-L198)). Delegating would make the two agree, but neither would match Spark. Spark's [`parseExtractField`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L4567-L4597) rejects `NANOSECOND` and `EPOCH`, and its `SECOND` is [`SecondWithFraction`](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L665-L669), a `DECIMAL(8, 6)` that keeps the fraction. Could we fix that in a separate PR, b
y rejecting the parts Spark rejects and returning Spark's type for `SECOND`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375415460
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375354681)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6375354681-3812-68qz6 6.12.105+ #1 SMP Sat Sep 5 07:57:24 UTC 2026 x86_64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384586255
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384376483)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.06 270.2±13.44ns ? ?/sec 1.00 254.2±6.00ns ? ?/sec
date_part_microsecond_date64_1000 1.07 271.1±14.99ns ? ?/sec 1.00 253.4±7.04ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 4.08 3.5±0.21µs ? ?/sec 1.00 861.4±54.42ns ? ?/sec
date_part_microsecond_interval_dt_1000 6.10 6.5±0.17µs ? ?/sec 1.00 1056.8±59.37ns ? ?/sec
date_part_microsecond_interval_mdn_1000 4.99 6.7±0.17µs ? ?/sec 1.00 1339.0±39.71ns ? ?/sec
date_part_microsecond_ms_1000 2.44 20.1±0.68µs ? ?/sec 1.00 8.2±0.22µs ? ?/sec
date_part_microsecond_ns_1000 2.42 19.1±0.37µs ? ?/sec 1.00 7.9±0.27µs ? ?/sec
date_part_microsecond_ns_nulls_1000 1.00 8.0±0.61µs ? ?/sec
date_part_microsecond_ns_tz_1000 2.82 86.3±1.31µs ? ?/sec 1.00 30.6±0.80µs ? ?/sec
date_part_microsecond_s_1000 1.53 9.3±0.19µs ? ?/sec 1.00 6.1±0.16µs ? ?/sec
date_part_microsecond_time32_ms_1000 3.63 4.0±0.13µs ? ?/sec 1.00 1111.3±65.24ns ? ?/sec
date_part_microsecond_time64_ns_1000 3.29 3.8±0.10µs ? ?/sec 1.00 1160.7±29.72ns ? ?/sec
date_part_microsecond_time64_ns_nulls_1000 1.00 2.1±0.13µs ? ?/sec
date_part_microsecond_us_1000 2.67 20.4±0.43µs ? ?/sec 1.00 7.7±0.27µs ? ?/sec
date_part_millisecond_date32_1000 1.06 269.2±14.45ns ? ?/sec 1.00 255.1±7.54ns ? ?/sec
date_part_millisecond_date64_1000 1.01 263.7±7.90ns ? ?/sec 1.00 261.6±15.45ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 3.44 3.3±0.13µs ? ?/sec 1.00 957.1±29.99ns ? ?/sec
date_part_millisecond_interval_dt_1000 8.06 6.4±0.17µs ? ?/sec 1.00 798.9±22.48ns ? ?/sec
date_part_millisecond_interval_mdn_1000 5.26 6.8±0.38µs ? ?/sec 1.00 1291.4±79.19ns ? ?/sec
date_part_millisecond_ms_1000 2.44 20.0±0.65µs ? ?/sec 1.00 8.2±0.22µs ? ?/sec
date_part_millisecond_ns_1000 2.47 19.3±0.54µs ? ?/sec 1.00 7.8±0.26µs ? ?/sec
date_part_millisecond_ns_nulls_1000 1.00 7.9±0.15µs ? ?/sec
date_part_millisecond_ns_tz_1000 2.80 87.8±3.58µs ? ?/sec 1.00 31.4±1.06µs ? ?/sec
date_part_millisecond_s_1000 1.51 9.2±0.16µs ? ?/sec 1.00 6.1±0.21µs ? ?/sec
date_part_millisecond_time32_ms_1000 4.84 4.0±0.06µs ? ?/sec 1.00 829.9±26.80ns ? ?/sec
date_part_millisecond_time64_ns_1000 3.31 3.8±0.14µs ? ?/sec 1.00 1159.6±19.68ns ? ?/sec
date_part_millisecond_time64_ns_nulls_1000 1.00 2.1±0.05µs ? ?/sec
date_part_millisecond_us_1000 2.63 20.5±0.72µs ? ?/sec 1.00 7.8±0.24µs ? ?/sec
date_part_nanosecond_date32_1000 1.06 287.5±16.84ns ? ?/sec 1.00 271.9±7.85ns ? ?/sec
date_part_nanosecond_date64_1000 1.03 279.9±6.67ns ? ?/sec 1.00 270.8±7.63ns ? ?/sec
date_part_nanosecond_duration_ns_1000 12.38 5.3±0.31µs ? ?/sec 1.00 427.5±13.49ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 16.66 7.8±0.31µs ? ?/sec 1.00 467.1±19.51ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 4.03 1721.6±58.50ns ? ?/sec 1.00 426.8±11.07ns ? ?/sec
date_part_nanosecond_interval_dt_1000 6.40 6.4±0.30µs ? ?/sec 1.00 993.8±30.14ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 8.26 6.5±0.26µs ? ?/sec 1.00 782.5±21.48ns ? ?/sec
date_part_nanosecond_ms_1000 2.25 18.6±0.70µs ? ?/sec 1.00 8.3±0.21µs ? ?/sec
date_part_nanosecond_ns_1000 2.26 17.8±0.76µs ? ?/sec 1.00 7.9±0.33µs ? ?/sec
date_part_nanosecond_ns_nulls_1000 1.00 8.0±0.38µs ? ?/sec
date_part_nanosecond_ns_tz_1000 2.75 85.7±3.14µs ? ?/sec 1.00 31.1±1.68µs ? ?/sec
date_part_nanosecond_s_1000 1.18 7.7±0.24µs ? ?/sec 1.00 6.5±0.23µs ? ?/sec
date_part_nanosecond_time32_ms_1000 2.20 2.5±0.08µs ? ?/sec 1.00 1128.0±58.58ns ? ?/sec
date_part_nanosecond_time64_ns_1000 2.69 2.3±0.12µs ? ?/sec 1.00 861.4±27.48ns ? ?/sec
date_part_nanosecond_time64_ns_nulls_1000 1.00 1357.4±29.22ns ? ?/sec
date_part_nanosecond_us_1000 2.42 19.0±0.60µs ? ?/sec 1.00 7.8±0.34µs ? ?/sec
date_part_second_date32_1000 1.02 273.5±5.12ns ? ?/sec 1.00 268.6±7.34ns ? ?/sec
date_part_second_date64_1000 1.02 9.6±0.57µs ? ?/sec 1.00 9.4±0.35µs ? ?/sec
date_part_second_ms_1000 1.00 9.3±0.24µs ? ?/sec 1.00 9.3±0.19µs ? ?/sec
date_part_second_ns_1000 1.00 8.8±0.24µs ? ?/sec 1.00 8.8±0.23µs ? ?/sec
date_part_second_s_1000 1.00 7.2±0.13µs ? ?/sec 1.02 7.3±0.23µs ? ?/sec
date_part_second_us_1000 1.01 9.4±0.17µs ? ?/sec 1.00 9.3±0.22µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 510.1s |
| Peak memory | 44.0 MiB |
| Avg memory | 32.7 MiB |
| CPU user | 605.3s |
| CPU sys | 0.4s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 580.1s |
| Peak memory | 45.1 MiB |
| Avg memory | 33.7 MiB |
| CPU user | 676.2s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367248751
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
+ // arrow's sub-second parts of intervals and durations already include
+ // the whole seconds
+ (
+ Interval(ArrowIntervalUnit::DayTime | ArrowIntervalUnit::MonthDayNano)
+ | Duration(_),
+ _,
+ ) => date_part(array, part)?,
Review Comment:
> Could this be `Interval(_) | Duration(_)`?
Done in [`date_part.rs`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/functions/src/datetime/date_part.rs#L545-L547). `nanosecond` fell through the same way, so the remaining [`query error`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1385-L1386) now checks for `Nanosecond does not support`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375351277
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
resources:
arch: amd64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375740311
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375349435)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.06 343.2±0.56ns ? ?/sec 1.00 322.8±4.68ns ? ?/sec
date_part_microsecond_date64_1000 1.06 344.9±1.16ns ? ?/sec 1.00 325.7±4.96ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 5.16 5.5±0.01µs ? ?/sec 1.00 1060.2±3.97ns ? ?/sec
date_part_microsecond_interval_dt_1000 8.75 7.5±0.06µs ? ?/sec 1.00 858.7±3.98ns ? ?/sec
date_part_microsecond_interval_mdn_1000 4.62 8.3±0.07µs ? ?/sec 1.00 1799.0±4.43ns ? ?/sec
date_part_microsecond_ms_1000 1.17 23.5±0.03µs ? ?/sec 1.00 20.2±0.03µs ? ?/sec
date_part_microsecond_ns_1000 1.17 21.8±0.03µs ? ?/sec 1.00 18.7±0.01µs ? ?/sec
date_part_microsecond_ns_tz_1000 1.09 94.7±0.50µs ? ?/sec 1.00 86.5±0.22µs ? ?/sec
date_part_microsecond_s_1000 1.27 10.4±0.01µs ? ?/sec 1.00 8.2±0.02µs ? ?/sec
date_part_microsecond_time32_ms_1000 2.35 6.0±0.01µs ? ?/sec 1.00 2.6±0.00µs ? ?/sec
date_part_microsecond_time64_ns_1000 2.18 6.0±0.12µs ? ?/sec 1.00 2.7±0.00µs ? ?/sec
date_part_microsecond_us_1000 1.16 22.3±0.02µs ? ?/sec 1.00 19.2±0.08µs ? ?/sec
date_part_millisecond_date32_1000 1.05 349.5±0.89ns ? ?/sec 1.00 332.5±3.50ns ? ?/sec
date_part_millisecond_date64_1000 1.08 356.7±3.84ns ? ?/sec 1.00 331.1±4.06ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 3.94 4.7±0.01µs ? ?/sec 1.00 1180.7±3.22ns ? ?/sec
date_part_millisecond_interval_dt_1000 9.22 7.5±0.07µs ? ?/sec 1.00 811.2±4.80ns ? ?/sec
date_part_millisecond_interval_mdn_1000 4.59 8.4±0.06µs ? ?/sec 1.00 1820.1±5.29ns ? ?/sec
date_part_millisecond_ms_1000 1.13 22.7±0.04µs ? ?/sec 1.00 20.1±0.04µs ? ?/sec
date_part_millisecond_ns_1000 1.11 21.0±0.02µs ? ?/sec 1.00 19.0±0.03µs ? ?/sec
date_part_millisecond_ns_tz_1000 1.06 93.2±0.26µs ? ?/sec 1.00 87.5±0.40µs ? ?/sec
date_part_millisecond_s_1000 1.29 10.5±0.01µs ? ?/sec 1.00 8.2±0.01µs ? ?/sec
date_part_millisecond_time32_ms_1000 2.17 5.3±0.01µs ? ?/sec 1.00 2.4±0.01µs ? ?/sec
date_part_millisecond_time64_ns_1000 1.83 5.2±0.01µs ? ?/sec 1.00 2.8±0.00µs ? ?/sec
date_part_millisecond_us_1000 1.12 21.6±0.02µs ? ?/sec 1.00 19.3±0.03µs ? ?/sec
date_part_nanosecond_date32_1000 1.04 379.3±0.65ns ? ?/sec 1.00 364.6±3.40ns ? ?/sec
date_part_nanosecond_date64_1000 1.07 381.9±0.87ns ? ?/sec 1.00 356.8±3.40ns ? ?/sec
date_part_nanosecond_duration_ns_1000 28.73 6.5±0.05µs ? ?/sec 1.00 227.4±3.11ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 32.18 8.4±0.09µs ? ?/sec 1.00 260.3±2.85ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 9.25 2.1±0.00µs ? ?/sec 1.00 226.2±3.32ns ? ?/sec
date_part_nanosecond_interval_dt_1000 9.74 7.1±0.05µs ? ?/sec 1.00 724.6±3.91ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 6.53 7.9±0.05µs ? ?/sec 1.00 1216.0±3.83ns ? ?/sec
date_part_nanosecond_ms_1000 1.01 20.2±0.09µs ? ?/sec 1.00 20.0±0.02µs ? ?/sec
date_part_nanosecond_ns_1000 1.00 18.5±0.03µs ? ?/sec 1.01 18.7±0.10µs ? ?/sec
date_part_nanosecond_ns_tz_1000 1.04 91.0±0.34µs ? ?/sec 1.00 87.6±0.24µs ? ?/sec
date_part_nanosecond_s_1000 1.00 8.2±0.01µs ? ?/sec 1.00 8.3±0.01µs ? ?/sec
date_part_nanosecond_time32_ms_1000 1.06 2.6±0.01µs ? ?/sec 1.00 2.5±0.00µs ? ?/sec
date_part_nanosecond_time64_ns_1000 1.00 2.6±0.01µs ? ?/sec 1.05 2.7±0.00µs ? ?/sec
date_part_nanosecond_us_1000 1.01 19.3±0.02µs ? ?/sec 1.00 19.1±0.08µs ? ?/sec
date_part_second_date32_1000 1.00 351.4±1.90ns ? ?/sec 1.00 351.6±3.42ns ? ?/sec
date_part_second_date64_1000 1.00 10.3±0.01µs ? ?/sec 1.00 10.3±0.01µs ? ?/sec
date_part_second_ms_1000 1.00 10.2±0.01µs ? ?/sec 1.00 10.2±0.01µs ? ?/sec
date_part_second_ns_1000 1.00 9.6±0.02µs ? ?/sec 1.00 9.6±0.01µs ? ?/sec
date_part_second_s_1000 1.01 7.7±0.01µs ? ?/sec 1.00 7.6±0.01µs ? ?/sec
date_part_second_us_1000 1.00 9.7±0.01µs ? ?/sec 1.02 9.9±0.03µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 675.1s |
| Peak memory | 34.8 MiB |
| Avg memory | 21.2 MiB |
| CPU user | 550.0s |
| CPU sys | 0.6s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 680.1s |
| Peak memory | 38.0 MiB |
| Avg memory | 23.4 MiB |
| CPU user | 558.7s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367250013
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
+ // arrow's sub-second parts of intervals and durations already include
+ // the whole seconds
+ (
+ Interval(ArrowIntervalUnit::DayTime | ArrowIntervalUnit::MonthDayNano)
+ | Duration(_),
+ _,
+ ) => date_part(array, part)?,
+ (_, DatePart::Millisecond) => {
+ seconds_with_fraction::<Int32Type, 1_000>(array, part)?
+ }
+ (_, DatePart::Microsecond) => {
+ seconds_with_fraction::<Int32Type, 1_000_000>(array, part)?
+ }
+ (_, DatePart::Nanosecond) => {
+ seconds_with_fraction::<Int64Type, 1_000_000_000>(array, part)?
+ }
+ (_, part) => return internal_err!("{part} is not a sub-second part"),
+ })
+}
+/// Combine arrow's seconds of the minute with its fraction of the second in
+/// `part` units, for types where arrow returns them separately (timestamps
+/// and times).
+fn seconds_with_fraction<O, const PER_SECOND: i64>(
+ array: &dyn Array,
+ part: DatePart,
+) -> Result<ArrayRef>
+where
+ O: ArrowPrimitiveType,
+ i64: AsPrimitive<O::Native>,
+{
let secs = date_part(array, DatePart::Second)?;
// This assumes array is primitive and not a dictionary
let secs = as_int32_array(secs.as_ref())?;
- let subsecs = date_part(array, DatePart::Nanosecond)?;
- let subsecs = as_int32_array(subsecs.as_ref())?;
-
- // Special case where there are no nulls.
- if subsecs.null_count() == 0 {
- let r: Int64Array = binary(secs, subsecs, |secs, subsecs| {
- (secs as i64) * 1_000_000_000 + (subsecs as i64)
- })?;
- Ok(Arc::new(r))
- } else {
- // Nulls in secs are preserved, nulls in subsecs are treated as zero to account for the case
- // where the number of nanoseconds overflows.
- let r: Int64Array = secs
- .iter()
- .zip(subsecs)
- .map(|(secs, subsecs)| {
- secs.map(|secs| {
- let subsecs = subsecs.unwrap_or(0);
- (secs as i64) * 1_000_000_000 + (subsecs as i64)
- })
- })
- .collect();
- Ok(Arc::new(r))
- }
+ let fraction = date_part(array, part)?;
Review Comment:
> `main` already used two passes, so a follow-up is fine.
I did it in this PR, in [`date_part.rs`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/functions/src/datetime/date_part.rs#L561-L695). Two things differ from your sketch:
- The timestamp helper takes arrow's converter for the unit (`timestamp_s_to_datetime` and so on), which arrow's `date_part` calls too. With `as_datetime::<T>` it measured about 1.5x slower, and `Timestamp(Second)` came out slower than before.
- Times use arithmetic with the same one-day range check as arrow's `date_part`, not `as_time`. `time64ns_to_time` computes `(v / NANOSECONDS) as u32`, so `2^32 + 5` seconds becomes 00:00:05 instead of `None` (apache/arrow-rs#11893). A [new query](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L952-L965) checks that out-of-range times stay NULL.
Over `c920a5b92`, `millisecond` on `Timestamp(ns)` went from 1.07x to 1.12x before this change to 2.18x to 2.56x, and `microsecond` on `Time64(ns)` from 1.53x to 2.24x to 3.23x to 4.77x, across the four bot runs linked in the description. The timezone case gained about as much as the others, 2.34x to 2.84x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375354681
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
resources:
arch: amd64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375349435
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375735224
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375351277)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.03 245.0±2.23ns ? ?/sec 1.00 236.9±2.53ns ? ?/sec
date_part_microsecond_date64_1000 1.04 245.6±1.22ns ? ?/sec 1.00 235.8±2.48ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 3.64 3.2±0.02µs ? ?/sec 1.00 867.2±10.56ns ? ?/sec
date_part_microsecond_interval_dt_1000 6.39 6.2±0.07µs ? ?/sec 1.00 977.4±16.36ns ? ?/sec
date_part_microsecond_interval_mdn_1000 5.83 6.5±0.15µs ? ?/sec 1.00 1117.8±14.44ns ? ?/sec
date_part_microsecond_ms_1000 1.07 19.5±0.11µs ? ?/sec 1.00 18.3±0.35µs ? ?/sec
date_part_microsecond_ns_1000 1.07 18.7±0.08µs ? ?/sec 1.00 17.5±0.12µs ? ?/sec
date_part_microsecond_ns_tz_1000 1.04 84.0±1.76µs ? ?/sec 1.00 81.1±0.67µs ? ?/sec
date_part_microsecond_s_1000 1.21 9.0±0.09µs ? ?/sec 1.00 7.5±0.05µs ? ?/sec
date_part_microsecond_time32_ms_1000 1.69 3.8±0.03µs ? ?/sec 1.00 2.3±0.01µs ? ?/sec
date_part_microsecond_time64_ns_1000 1.53 3.6±0.02µs ? ?/sec 1.00 2.4±0.05µs ? ?/sec
date_part_microsecond_us_1000 1.07 19.8±0.11µs ? ?/sec 1.00 18.6±0.13µs ? ?/sec
date_part_millisecond_date32_1000 1.04 245.5±1.85ns ? ?/sec 1.00 235.1±4.28ns ? ?/sec
date_part_millisecond_date64_1000 1.04 245.3±2.09ns ? ?/sec 1.00 235.1±2.18ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 3.83 3.1±0.03µs ? ?/sec 1.00 813.7±6.18ns ? ?/sec
date_part_millisecond_interval_dt_1000 8.29 6.2±0.03µs ? ?/sec 1.00 750.3±6.78ns ? ?/sec
date_part_millisecond_interval_mdn_1000 5.14 6.5±0.08µs ? ?/sec 1.00 1273.2±27.35ns ? ?/sec
date_part_millisecond_ms_1000 1.06 19.5±0.10µs ? ?/sec 1.00 18.4±0.19µs ? ?/sec
date_part_millisecond_ns_1000 1.08 18.8±0.13µs ? ?/sec 1.00 17.4±0.14µs ? ?/sec
date_part_millisecond_ns_tz_1000 1.04 83.8±0.65µs ? ?/sec 1.00 80.5±0.92µs ? ?/sec
date_part_millisecond_s_1000 1.20 9.0±0.07µs ? ?/sec 1.00 7.5±0.05µs ? ?/sec
date_part_millisecond_time32_ms_1000 1.91 3.8±0.04µs ? ?/sec 1.00 2.0±0.03µs ? ?/sec
date_part_millisecond_time64_ns_1000 1.58 3.6±0.02µs ? ?/sec 1.00 2.3±0.03µs ? ?/sec
date_part_millisecond_us_1000 1.07 19.9±0.13µs ? ?/sec 1.00 18.6±0.14µs ? ?/sec
date_part_nanosecond_date32_1000 1.03 258.5±1.57ns ? ?/sec 1.00 250.0±1.85ns ? ?/sec
date_part_nanosecond_date64_1000 1.04 260.1±2.83ns ? ?/sec 1.00 249.0±1.03ns ? ?/sec
date_part_nanosecond_duration_ns_1000 29.10 4.9±0.03µs ? ?/sec 1.00 169.9±2.19ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 41.62 7.9±0.04µs ? ?/sec 1.00 190.7±3.09ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 9.76 1655.0±7.95ns ? ?/sec 1.00 169.5±1.64ns ? ?/sec
date_part_nanosecond_interval_dt_1000 6.20 5.9±0.04µs ? ?/sec 1.00 955.6±10.00ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 8.72 6.2±0.05µs ? ?/sec 1.00 716.7±8.71ns ? ?/sec
date_part_nanosecond_ms_1000 1.00 18.1±0.20µs ? ?/sec 1.02 18.4±0.28µs ? ?/sec
date_part_nanosecond_ns_1000 1.00 17.3±0.16µs ? ?/sec 1.02 17.6±0.14µs ? ?/sec
date_part_nanosecond_ns_tz_1000 1.01 82.2±0.33µs ? ?/sec 1.00 81.0±0.58µs ? ?/sec
date_part_nanosecond_s_1000 1.00 7.5±0.05µs ? ?/sec 1.02 7.7±0.10µs ? ?/sec
date_part_nanosecond_time32_ms_1000 1.00 2.4±0.02µs ? ?/sec 1.03 2.4±0.02µs ? ?/sec
date_part_nanosecond_time64_ns_1000 1.00 2.1±0.01µs ? ?/sec 1.03 2.2±0.04µs ? ?/sec
date_part_nanosecond_us_1000 1.00 18.3±0.13µs ? ?/sec 1.02 18.8±0.22µs ? ?/sec
date_part_second_date32_1000 1.08 254.6±3.25ns ? ?/sec 1.00 235.8±2.20ns ? ?/sec
date_part_second_date64_1000 1.00 9.0±0.06µs ? ?/sec 1.02 9.2±0.08µs ? ?/sec
date_part_second_ms_1000 1.00 9.0±0.06µs ? ?/sec 1.02 9.2±0.10µs ? ?/sec
date_part_second_ns_1000 1.00 8.7±0.06µs ? ?/sec 1.00 8.7±0.07µs ? ?/sec
date_part_second_s_1000 1.00 7.1±0.10µs ? ?/sec 1.01 7.1±0.03µs ? ?/sec
date_part_second_us_1000 1.00 9.2±0.06µs ? ?/sec 1.01 9.3±0.06µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 650.2s |
| Peak memory | 102.7 MiB |
| Avg memory | 25.5 MiB |
| CPU user | 587.0s |
| CPU sys | 0.4s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 650.2s |
| Peak memory | 42.3 MiB |
| Avg memory | 25.7 MiB |
| CPU user | 592.1s |
| CPU sys | 0.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6376168507
I ran `cargo xtask ci step test extended` locally to see if any tests there were angry about behavior changes, but didn't spot any failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384372592
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367247439
##########
datafusion/sqllogictest/test_files/datetime/date_part.slt:
##########
@@ -1289,6 +1419,124 @@ SELECT extract(day from arrow_cast(NULL, 'Duration(Second)'))
----
NULL
+# nanosecond on a duration is its total length in nanoseconds
+query IIII
+SELECT extract(nanosecond from arrow_cast(1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(1500000, 'Duration(Microsecond)')),
+ extract(nanosecond from arrow_cast(1500, 'Duration(Millisecond)')),
+ extract(nanosecond from arrow_cast(2, 'Duration(Second)'))
+----
+1500000000 1500000000 1500000000 2000000000
+
+# around i32::MAX nanoseconds
+query III
+SELECT extract(nanosecond from arrow_cast(2147483647, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(2147483648, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(62500000000, 'Duration(Nanosecond)'))
+----
+2147483647 2147483648 62500000000
+
+query II
+SELECT extract(nanosecond from arrow_cast(-1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(-62500, 'Duration(Millisecond)'))
+----
+-1500000000 -62500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast(column1, 'Duration(Millisecond)'))
+FROM (VALUES (1500), (NULL), (62500), (NULL), (-2))
+----
+1500000000
+NULL
+62500000000
+NULL
+-2000000
+
+query II
+SELECT extract(nanosecond from arrow_cast(9223372036854775807, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(-9223372036854775808, 'Duration(Nanosecond)'))
+----
+9223372036854775807 -9223372036854775808
+
+# values that overflow are NULL, like the other parts of a duration
+query II
+SELECT extract(nanosecond from arrow_cast(column1, 'Duration(Second)')),
+ extract(nanosecond from arrow_cast(column2, 'Duration(Millisecond)'))
+FROM (VALUES (9223372036, -9223372036854), (9223372037, -9223372036855), (NULL, NULL))
+----
+9223372036000000000 -9223372036854000000
+NULL NULL
+NULL NULL
+
+# millisecond and microsecond on a duration are its total length in that unit
+query II
+SELECT extract(millisecond from arrow_cast(5500000000, 'Duration(Nanosecond)')),
+ extract(microsecond from arrow_cast(5500000000, 'Duration(Nanosecond)'))
+----
+5500 5500000
+
+# truncates toward zero, like the other parts of a duration
+query III
+SELECT extract(millisecond from arrow_cast(-1500, 'Duration(Microsecond)')),
+ extract(millisecond from arrow_cast(-1500000, 'Duration(Nanosecond)')),
+ extract(second from arrow_cast(-1500, 'Duration(Millisecond)'))
+----
+-1 -1 -1
+
+query I
+SELECT extract(microsecond from arrow_cast(column1, 'Duration(Nanosecond)'))
+FROM (VALUES (5500000000), (NULL), (-62500000000), (NULL))
+----
+5500000
+NULL
+-62500000
+NULL
+
+# around i32::MAX
+query II
+SELECT extract(millisecond from arrow_cast(2147483647, 'Duration(Millisecond)')),
+ extract(microsecond from arrow_cast(2147, 'Duration(Second)'))
+----
+2147483647 2147000000
+
+# values that overflow Int32 are NULL, like the other parts of a duration
+query II
+SELECT extract(millisecond from arrow_cast(column1, 'Duration(Millisecond)')),
+ extract(microsecond from arrow_cast(column2, 'Duration(Second)'))
+FROM (VALUES (2147483648, 2148), (-2147483649, -2148), (NULL, NULL))
+----
+NULL NULL
+NULL NULL
+NULL NULL
+
+# Reproducer from https://github.com/apache/datafusion/issues/14738: 2.4e9 microseconds don't fit in Int32
+query I
+SELECT date_part('microsecond', timestamp '1970-01-01T00:40:00' - timestamp '1970-01-01T00:00:00')
+----
Review Comment:
> Should this be `Part of #14738`, with the semantics question left open? Or is NULL the agreed outcome?
I changed it to `Part of #14738` and left a comment on the issue. The reproducer's [comment](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1608-L1614) now says why Postgres returns 0 and this returns NULL. On the [dev list](https://lists.apache.org/thread.html/90cqgs9dotp39tcwr5ophvb2ftvjc43p), `@alamb` agreed with NULL for now and suggested following Postgres where we can. I'd leave the rest to the issue, since adopting Postgres's meaning would change every `Duration` part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384582980
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384374967)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.08 355.1±1.89ns ? ?/sec 1.00 330.0±0.65ns ? ?/sec
date_part_microsecond_date64_1000 1.11 365.7±1.83ns ? ?/sec 1.00 330.6±0.55ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 5.23 5.5±0.01µs ? ?/sec 1.00 1053.2±3.40ns ? ?/sec
date_part_microsecond_interval_dt_1000 8.72 7.5±0.08µs ? ?/sec 1.00 855.1±2.40ns ? ?/sec
date_part_microsecond_interval_mdn_1000 4.59 8.3±0.08µs ? ?/sec 1.00 1811.4±3.29ns ? ?/sec
date_part_microsecond_ms_1000 2.31 23.5±0.02µs ? ?/sec 1.00 10.2±0.01µs ? ?/sec
date_part_microsecond_ns_1000 2.26 22.0±0.02µs ? ?/sec 1.00 9.7±0.01µs ? ?/sec
date_part_microsecond_ns_nulls_1000 1.00 9.6±0.09µs ? ?/sec
date_part_microsecond_ns_tz_1000 2.34 95.1±0.39µs ? ?/sec 1.00 40.7±0.12µs ? ?/sec
date_part_microsecond_s_1000 1.44 10.5±0.01µs ? ?/sec 1.00 7.3±0.00µs ? ?/sec
date_part_microsecond_time32_ms_1000 5.13 6.0±0.01µs ? ?/sec 1.00 1166.4±2.32ns ? ?/sec
date_part_microsecond_time64_ns_1000 4.77 6.0±0.01µs ? ?/sec 1.00 1258.2±2.61ns ? ?/sec
date_part_microsecond_time64_ns_nulls_1000 1.00 1638.3±2.16ns ? ?/sec
date_part_microsecond_us_1000 2.40 22.4±0.09µs ? ?/sec 1.00 9.3±0.00µs ? ?/sec
date_part_millisecond_date32_1000 1.08 358.4±1.66ns ? ?/sec 1.00 331.5±0.65ns ? ?/sec
date_part_millisecond_date64_1000 1.08 354.9±2.39ns ? ?/sec 1.00 329.9±0.97ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 4.47 4.7±0.00µs ? ?/sec 1.00 1045.9±2.93ns ? ?/sec
date_part_millisecond_interval_dt_1000 9.20 7.4±0.08µs ? ?/sec 1.00 807.8±2.15ns ? ?/sec
date_part_millisecond_interval_mdn_1000 4.63 8.3±0.08µs ? ?/sec 1.00 1793.4±4.64ns ? ?/sec
date_part_millisecond_ms_1000 2.33 22.8±0.03µs ? ?/sec 1.00 9.8±0.01µs ? ?/sec
date_part_millisecond_ns_1000 2.18 21.0±0.02µs ? ?/sec 1.00 9.6±0.00µs ? ?/sec
date_part_millisecond_ns_nulls_1000 1.00 9.3±0.02µs ? ?/sec
date_part_millisecond_ns_tz_1000 2.42 94.4±0.50µs ? ?/sec 1.00 39.0±0.66µs ? ?/sec
date_part_millisecond_s_1000 1.48 10.6±0.01µs ? ?/sec 1.00 7.2±0.00µs ? ?/sec
date_part_millisecond_time32_ms_1000 4.68 5.2±0.01µs ? ?/sec 1.00 1115.6±2.69ns ? ?/sec
date_part_millisecond_time64_ns_1000 4.07 5.1±0.00µs ? ?/sec 1.00 1265.2±1.70ns ? ?/sec
date_part_millisecond_time64_ns_nulls_1000 1.00 1643.6±2.90ns ? ?/sec
date_part_millisecond_us_1000 2.22 21.5±0.01µs ? ?/sec 1.00 9.7±0.03µs ? ?/sec
date_part_nanosecond_date32_1000 1.06 377.8±1.45ns ? ?/sec 1.00 355.5±0.82ns ? ?/sec
date_part_nanosecond_date64_1000 1.09 383.4±2.00ns ? ?/sec 1.00 352.0±0.70ns ? ?/sec
date_part_nanosecond_duration_ns_1000 12.20 6.5±0.05µs ? ?/sec 1.00 534.7±4.99ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 14.00 8.5±0.06µs ? ?/sec 1.00 603.9±3.19ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 3.94 2.1±0.00µs ? ?/sec 1.00 537.2±3.53ns ? ?/sec
date_part_nanosecond_interval_dt_1000 9.58 7.0±0.06µs ? ?/sec 1.00 734.1±2.89ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 6.48 7.9±0.06µs ? ?/sec 1.00 1214.1±2.92ns ? ?/sec
date_part_nanosecond_ms_1000 1.97 20.2±0.03µs ? ?/sec 1.00 10.3±0.02µs ? ?/sec
date_part_nanosecond_ns_1000 1.98 18.5±0.02µs ? ?/sec 1.00 9.4±0.01µs ? ?/sec
date_part_nanosecond_ns_nulls_1000 1.00 9.2±0.01µs ? ?/sec
date_part_nanosecond_ns_tz_1000 2.46 91.9±0.36µs ? ?/sec 1.00 37.3±0.34µs ? ?/sec
date_part_nanosecond_s_1000 1.15 8.3±0.01µs ? ?/sec 1.00 7.2±0.00µs ? ?/sec
date_part_nanosecond_time32_ms_1000 2.02 2.6±0.00µs ? ?/sec 1.00 1270.7±2.18ns ? ?/sec
date_part_nanosecond_time64_ns_1000 2.23 2.6±0.00µs ? ?/sec 1.00 1166.5±2.52ns ? ?/sec
date_part_nanosecond_time64_ns_nulls_1000 1.00 1579.1±1.71ns ? ?/sec
date_part_nanosecond_us_1000 1.98 19.1±0.04µs ? ?/sec 1.00 9.6±0.02µs ? ?/sec
date_part_second_date32_1000 1.03 363.6±2.77ns ? ?/sec 1.00 353.9±2.25ns ? ?/sec
date_part_second_date64_1000 1.00 10.2±0.01µs ? ?/sec 1.00 10.2±0.01µs ? ?/sec
date_part_second_ms_1000 1.00 10.2±0.01µs ? ?/sec 1.01 10.3±0.01µs ? ?/sec
date_part_second_ns_1000 1.00 9.6±0.01µs ? ?/sec 1.00 9.6±0.01µs ? ?/sec
date_part_second_s_1000 1.00 7.7±0.01µs ? ?/sec 1.01 7.7±0.04µs ? ?/sec
date_part_second_us_1000 1.01 9.9±0.01µs ? ?/sec 1.00 9.8±0.02µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 500.1s |
| Peak memory | 38.9 MiB |
| Avg memory | 30.8 MiB |
| CPU user | 543.5s |
| CPU sys | 0.6s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 600.1s |
| Peak memory | 36.6 MiB |
| Avg memory | 27.7 MiB |
| CPU user | 627.2s |
| CPU sys | 0.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367250542
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -891,6 +866,38 @@ mod tests {
}
}
+ /// `epoch` is NULL only for a NULL input, but the integer parts can be NULL
+ /// for a non-null input, so they are always nullable.
+ #[test]
+ fn return_field_nullability() {
Review Comment:
> Could this move to the slt with the existing `DESCRIBE` idiom (see `describe.slt`)?
Done in [`date_part.slt`](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1638-L1654), and the unit test is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384403936
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384374967)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6384374967-3832-485df 6.12.105+ #1 SMP Sat Sep 5 07:54:11 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384376483
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
resources:
arch: amd64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384374967
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384577309
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384373752)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.04 268.9±9.75ns ? ?/sec 1.00 259.3±10.84ns ? ?/sec
date_part_microsecond_date64_1000 1.00 264.2±12.20ns ? ?/sec 1.00 263.0±14.22ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 3.83 3.6±0.13µs ? ?/sec 1.00 927.9±26.63ns ? ?/sec
date_part_microsecond_interval_dt_1000 6.62 6.5±0.20µs ? ?/sec 1.00 974.9±18.27ns ? ?/sec
date_part_microsecond_interval_mdn_1000 5.42 6.7±0.36µs ? ?/sec 1.00 1244.1±27.64ns ? ?/sec
date_part_microsecond_ms_1000 2.45 20.3±0.66µs ? ?/sec 1.00 8.3±0.29µs ? ?/sec
date_part_microsecond_ns_1000 2.52 20.0±1.13µs ? ?/sec 1.00 8.0±0.33µs ? ?/sec
date_part_microsecond_ns_nulls_1000 1.00 7.8±0.30µs ? ?/sec
date_part_microsecond_ns_tz_1000 2.48 89.1±2.90µs ? ?/sec 1.00 36.0±0.62µs ? ?/sec
date_part_microsecond_s_1000 1.51 9.4±0.24µs ? ?/sec 1.00 6.2±0.30µs ? ?/sec
date_part_microsecond_time32_ms_1000 3.49 4.0±0.08µs ? ?/sec 1.00 1148.6±39.84ns ? ?/sec
date_part_microsecond_time64_ns_1000 3.23 3.8±0.08µs ? ?/sec 1.00 1162.9±21.93ns ? ?/sec
date_part_microsecond_time64_ns_nulls_1000 1.00 1583.0±80.51ns ? ?/sec
date_part_microsecond_us_1000 2.72 21.0±1.10µs ? ?/sec 1.00 7.7±0.22µs ? ?/sec
date_part_millisecond_date32_1000 1.08 274.0±18.99ns ? ?/sec 1.00 253.7±6.97ns ? ?/sec
date_part_millisecond_date64_1000 1.05 266.1±7.57ns ? ?/sec 1.00 254.2±9.78ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 4.15 3.5±0.08µs ? ?/sec 1.00 851.5±14.40ns ? ?/sec
date_part_millisecond_interval_dt_1000 8.10 6.4±0.18µs ? ?/sec 1.00 796.1±25.92ns ? ?/sec
date_part_millisecond_interval_mdn_1000 5.60 7.0±0.54µs ? ?/sec 1.00 1254.0±44.60ns ? ?/sec
date_part_millisecond_ms_1000 2.47 20.4±0.86µs ? ?/sec 1.00 8.2±0.36µs ? ?/sec
date_part_millisecond_ns_1000 2.56 19.8±0.49µs ? ?/sec 1.00 7.7±0.14µs ? ?/sec
date_part_millisecond_ns_nulls_1000 1.00 7.9±0.16µs ? ?/sec
date_part_millisecond_ns_tz_1000 2.50 89.2±2.60µs ? ?/sec 1.00 35.6±1.22µs ? ?/sec
date_part_millisecond_s_1000 1.56 9.5±0.23µs ? ?/sec 1.00 6.1±0.20µs ? ?/sec
date_part_millisecond_time32_ms_1000 4.80 4.0±0.11µs ? ?/sec 1.00 841.0±39.33ns ? ?/sec
date_part_millisecond_time64_ns_1000 3.43 3.8±0.07µs ? ?/sec 1.00 1099.3±24.11ns ? ?/sec
date_part_millisecond_time64_ns_nulls_1000 1.00 1495.1±39.46ns ? ?/sec
date_part_millisecond_us_1000 2.66 20.7±0.61µs ? ?/sec 1.00 7.8±0.20µs ? ?/sec
date_part_nanosecond_date32_1000 1.03 280.0±12.11ns ? ?/sec 1.00 272.0±12.29ns ? ?/sec
date_part_nanosecond_date64_1000 1.02 282.0±11.55ns ? ?/sec 1.00 276.4±15.40ns ? ?/sec
date_part_nanosecond_duration_ns_1000 13.12 5.7±0.25µs ? ?/sec 1.00 435.8±24.80ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 17.23 8.0±0.26µs ? ?/sec 1.00 465.9±17.48ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 4.87 2.1±0.07µs ? ?/sec 1.00 423.1±11.60ns ? ?/sec
date_part_nanosecond_interval_dt_1000 6.31 6.3±0.18µs ? ?/sec 1.00 995.5±17.77ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 8.34 6.6±0.34µs ? ?/sec 1.00 785.4±15.79ns ? ?/sec
date_part_nanosecond_ms_1000 2.24 18.7±0.44µs ? ?/sec 1.00 8.4±0.44µs ? ?/sec
date_part_nanosecond_ns_1000 2.33 18.3±0.67µs ? ?/sec 1.00 7.9±0.17µs ? ?/sec
date_part_nanosecond_ns_nulls_1000 1.00 7.7±0.23µs ? ?/sec
date_part_nanosecond_ns_tz_1000 2.39 87.0±1.66µs ? ?/sec 1.00 36.4±1.63µs ? ?/sec
date_part_nanosecond_s_1000 1.23 7.9±0.20µs ? ?/sec 1.00 6.4±0.25µs ? ?/sec
date_part_nanosecond_time32_ms_1000 2.22 2.5±0.10µs ? ?/sec 1.00 1116.6±44.33ns ? ?/sec
date_part_nanosecond_time64_ns_1000 2.48 2.2±0.05µs ? ?/sec 1.00 897.8±32.65ns ? ?/sec
date_part_nanosecond_time64_ns_nulls_1000 1.00 1306.5±48.12ns ? ?/sec
date_part_nanosecond_us_1000 2.41 19.1±0.35µs ? ?/sec 1.00 7.9±0.35µs ? ?/sec
date_part_second_date32_1000 1.00 265.7±14.61ns ? ?/sec 1.03 274.4±15.57ns ? ?/sec
date_part_second_date64_1000 1.00 9.5±0.34µs ? ?/sec 1.01 9.6±0.30µs ? ?/sec
date_part_second_ms_1000 1.00 9.6±0.37µs ? ?/sec 1.00 9.6±0.34µs ? ?/sec
date_part_second_ns_1000 1.00 9.2±0.25µs ? ?/sec 1.02 9.3±0.28µs ? ?/sec
date_part_second_s_1000 1.02 7.6±0.26µs ? ?/sec 1.00 7.5±0.13µs ? ?/sec
date_part_second_us_1000 1.00 9.6±0.28µs ? ?/sec 1.03 9.9±0.33µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 515.1s |
| Peak memory | 42.0 MiB |
| Avg memory | 32.7 MiB |
| CPU user | 609.3s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 580.1s |
| Peak memory | 58.3 MiB |
| Avg memory | 33.1 MiB |
| CPU user | 676.5s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375414702
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375351277)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6375351277-3810-nwzn8 6.12.105+ #1 SMP Sat Sep 5 07:57:24 UTC 2026 x86_64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375408182
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375352980)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6375352980-3811-5q2pc 6.12.105+ #1 SMP Sat Sep 5 07:54:11 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375352980
run benchmark datetime_expressions
```yaml
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: c920a5b92df0172b23026c3773dc7cd0fbdb8ce6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6375763611
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6375354681)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (adab0854485d64a6d8c9922a47ce631bbef05178) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..adab0854485d64a6d8c9922a47ce631bbef05178)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.07 308.1±6.40ns ? ?/sec 1.00 287.0±8.12ns ? ?/sec
date_part_microsecond_date64_1000 1.07 309.0±8.24ns ? ?/sec 1.00 287.8±8.71ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 3.50 3.8±0.18µs ? ?/sec 1.00 1083.6±8.25ns ? ?/sec
date_part_microsecond_interval_dt_1000 5.97 7.8±0.19µs ? ?/sec 1.00 1306.3±26.01ns ? ?/sec
date_part_microsecond_interval_mdn_1000 5.22 8.1±0.21µs ? ?/sec 1.00 1559.9±32.86ns ? ?/sec
date_part_microsecond_ms_1000 1.09 24.6±0.62µs ? ?/sec 1.00 22.6±0.58µs ? ?/sec
date_part_microsecond_ns_1000 1.08 23.4±0.63µs ? ?/sec 1.00 21.7±0.50µs ? ?/sec
date_part_microsecond_ns_tz_1000 1.00 105.5±1.83µs ? ?/sec 1.00 105.7±2.55µs ? ?/sec
date_part_microsecond_s_1000 1.22 11.3±0.56µs ? ?/sec 1.00 9.3±0.24µs ? ?/sec
date_part_microsecond_time32_ms_1000 1.61 4.9±0.12µs ? ?/sec 1.00 3.0±0.06µs ? ?/sec
date_part_microsecond_time64_ns_1000 1.61 4.7±0.09µs ? ?/sec 1.00 2.9±0.06µs ? ?/sec
date_part_microsecond_us_1000 1.08 24.8±0.79µs ? ?/sec 1.00 23.0±0.93µs ? ?/sec
date_part_millisecond_date32_1000 1.05 307.1±8.48ns ? ?/sec 1.00 293.8±7.15ns ? ?/sec
date_part_millisecond_date64_1000 1.00 300.4±8.25ns ? ?/sec 1.03 309.7±8.91ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 3.86 4.0±0.12µs ? ?/sec 1.00 1024.5±26.89ns ? ?/sec
date_part_millisecond_interval_dt_1000 7.96 7.6±0.27µs ? ?/sec 1.00 950.1±36.13ns ? ?/sec
date_part_millisecond_interval_mdn_1000 5.23 7.8±0.27µs ? ?/sec 1.00 1491.1±61.65ns ? ?/sec
date_part_millisecond_ms_1000 1.08 24.0±0.87µs ? ?/sec 1.00 22.3±0.50µs ? ?/sec
date_part_millisecond_ns_1000 1.07 23.2±0.38µs ? ?/sec 1.00 21.6±0.52µs ? ?/sec
date_part_millisecond_ns_tz_1000 1.02 105.9±3.06µs ? ?/sec 1.00 104.0±3.19µs ? ?/sec
date_part_millisecond_s_1000 1.21 10.9±0.42µs ? ?/sec 1.00 9.0±0.33µs ? ?/sec
date_part_millisecond_time32_ms_1000 1.91 4.8±0.10µs ? ?/sec 1.00 2.5±0.10µs ? ?/sec
date_part_millisecond_time64_ns_1000 1.60 4.5±0.08µs ? ?/sec 1.00 2.8±0.09µs ? ?/sec
date_part_millisecond_us_1000 1.11 24.6±0.53µs ? ?/sec 1.00 22.2±0.97µs ? ?/sec
date_part_nanosecond_date32_1000 1.07 339.0±7.36ns ? ?/sec 1.00 317.6±11.01ns ? ?/sec
date_part_nanosecond_date64_1000 1.07 339.8±6.07ns ? ?/sec 1.00 317.2±10.93ns ? ?/sec
date_part_nanosecond_duration_ns_1000 26.71 6.0±0.34µs ? ?/sec 1.00 225.7±10.79ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 36.13 9.0±0.36µs ? ?/sec 1.00 250.4±6.24ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 8.99 2.0±0.11µs ? ?/sec 1.00 223.2±5.16ns ? ?/sec
date_part_nanosecond_interval_dt_1000 5.97 7.3±0.32µs ? ?/sec 1.00 1220.9±21.83ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 7.55 7.3±0.29µs ? ?/sec 1.00 971.9±17.34ns ? ?/sec
date_part_nanosecond_ms_1000 1.00 22.4±0.51µs ? ?/sec 1.00 22.3±0.78µs ? ?/sec
date_part_nanosecond_ns_1000 1.00 21.3±0.77µs ? ?/sec 1.00 21.3±0.60µs ? ?/sec
date_part_nanosecond_ns_tz_1000 1.00 101.2±2.58µs ? ?/sec 1.10 110.9±1.84µs ? ?/sec
date_part_nanosecond_s_1000 1.02 9.6±0.20µs ? ?/sec 1.00 9.3±0.22µs ? ?/sec
date_part_nanosecond_time32_ms_1000 1.00 3.0±0.09µs ? ?/sec 1.01 3.0±0.07µs ? ?/sec
date_part_nanosecond_time64_ns_1000 1.00 2.7±0.07µs ? ?/sec 1.00 2.7±0.05µs ? ?/sec
date_part_nanosecond_us_1000 1.03 22.7±0.41µs ? ?/sec 1.00 22.0±0.99µs ? ?/sec
date_part_second_date32_1000 1.01 318.0±11.02ns ? ?/sec 1.00 314.2±15.50ns ? ?/sec
date_part_second_date64_1000 1.05 11.3±0.25µs ? ?/sec 1.00 10.8±0.58µs ? ?/sec
date_part_second_ms_1000 1.05 11.3±0.24µs ? ?/sec 1.00 10.8±0.53µs ? ?/sec
date_part_second_ns_1000 1.06 10.7±0.23µs ? ?/sec 1.00 10.1±0.51µs ? ?/sec
date_part_second_s_1000 1.00 8.5±0.33µs ? ?/sec 1.01 8.6±0.21µs ? ?/sec
date_part_second_us_1000 1.03 11.3±0.34µs ? ?/sec 1.00 11.0±0.39µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 710.2s |
| Peak memory | 44.4 MiB |
| Avg memory | 25.0 MiB |
| CPU user | 644.3s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 705.2s |
| Peak memory | 46.0 MiB |
| Avg memory | 24.2 MiB |
| CPU user | 653.4s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384580919
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384372592)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD fix-extract-nanosecond-duration-interval
----- ---- ----------------------------------------
date_part_microsecond_date32_1000 1.02 339.2±1.59ns ? ?/sec 1.00 331.4±2.89ns ? ?/sec
date_part_microsecond_date64_1000 1.07 342.5±1.84ns ? ?/sec 1.00 320.0±3.19ns ? ?/sec
date_part_microsecond_duration_ns_small_1000 4.97 5.5±0.00µs ? ?/sec 1.00 1099.1±4.37ns ? ?/sec
date_part_microsecond_interval_dt_1000 8.64 7.5±0.06µs ? ?/sec 1.00 864.7±2.67ns ? ?/sec
date_part_microsecond_interval_mdn_1000 4.62 8.3±0.06µs ? ?/sec 1.00 1803.7±4.49ns ? ?/sec
date_part_microsecond_ms_1000 2.33 23.7±0.11µs ? ?/sec 1.00 10.1±0.01µs ? ?/sec
date_part_microsecond_ns_1000 2.26 21.9±0.02µs ? ?/sec 1.00 9.7±0.01µs ? ?/sec
date_part_microsecond_ns_nulls_1000 1.00 9.5±0.01µs ? ?/sec
date_part_microsecond_ns_tz_1000 2.84 94.1±0.38µs ? ?/sec 1.00 33.1±0.16µs ? ?/sec
date_part_microsecond_s_1000 1.45 10.4±0.01µs ? ?/sec 1.00 7.2±0.90µs ? ?/sec
date_part_microsecond_time32_ms_1000 4.69 6.0±0.01µs ? ?/sec 1.00 1269.5±2.79ns ? ?/sec
date_part_microsecond_time64_ns_1000 4.58 5.9±0.01µs ? ?/sec 1.00 1296.3±3.35ns ? ?/sec
date_part_microsecond_time64_ns_nulls_1000 1.00 1640.8±1.85ns ? ?/sec
date_part_microsecond_us_1000 2.42 22.5±0.01µs ? ?/sec 1.00 9.3±0.00µs ? ?/sec
date_part_millisecond_date32_1000 1.06 353.4±1.83ns ? ?/sec 1.00 333.3±2.83ns ? ?/sec
date_part_millisecond_date64_1000 1.11 356.4±2.17ns ? ?/sec 1.00 322.1±3.72ns ? ?/sec
date_part_millisecond_duration_ns_small_1000 3.93 4.7±0.01µs ? ?/sec 1.00 1186.1±4.67ns ? ?/sec
date_part_millisecond_interval_dt_1000 9.37 7.5±0.06µs ? ?/sec 1.00 796.5±3.26ns ? ?/sec
date_part_millisecond_interval_mdn_1000 4.59 8.3±0.06µs ? ?/sec 1.00 1813.0±4.17ns ? ?/sec
date_part_millisecond_ms_1000 2.30 22.8±0.02µs ? ?/sec 1.00 9.9±0.01µs ? ?/sec
date_part_millisecond_ns_1000 2.19 21.0±0.02µs ? ?/sec 1.00 9.6±0.01µs ? ?/sec
date_part_millisecond_ns_nulls_1000 1.00 9.4±0.01µs ? ?/sec
date_part_millisecond_ns_tz_1000 2.61 93.6±0.63µs ? ?/sec 1.00 35.9±0.20µs ? ?/sec
date_part_millisecond_s_1000 1.47 10.5±0.02µs ? ?/sec 1.00 7.1±0.01µs ? ?/sec
date_part_millisecond_time32_ms_1000 4.56 5.3±0.01µs ? ?/sec 1.00 1152.3±4.59ns ? ?/sec
date_part_millisecond_time64_ns_1000 4.06 5.1±0.01µs ? ?/sec 1.00 1264.2±4.50ns ? ?/sec
date_part_millisecond_time64_ns_nulls_1000 1.00 1660.3±2.21ns ? ?/sec
date_part_millisecond_us_1000 2.23 21.6±0.03µs ? ?/sec 1.00 9.7±0.01µs ? ?/sec
date_part_nanosecond_date32_1000 1.05 375.2±2.49ns ? ?/sec 1.00 357.7±3.98ns ? ?/sec
date_part_nanosecond_date64_1000 1.06 378.2±1.62ns ? ?/sec 1.00 357.0±2.99ns ? ?/sec
date_part_nanosecond_duration_ns_1000 12.11 6.6±0.04µs ? ?/sec 1.00 541.9±5.29ns ? ?/sec
date_part_nanosecond_duration_ns_nulls_1000 13.96 8.4±0.08µs ? ?/sec 1.00 598.8±3.85ns ? ?/sec
date_part_nanosecond_duration_ns_small_1000 3.86 2.1±0.01µs ? ?/sec 1.00 540.8±5.60ns ? ?/sec
date_part_nanosecond_interval_dt_1000 9.60 7.0±0.04µs ? ?/sec 1.00 730.9±3.69ns ? ?/sec
date_part_nanosecond_interval_mdn_1000 6.47 7.9±0.04µs ? ?/sec 1.00 1218.8±4.86ns ? ?/sec
date_part_nanosecond_ms_1000 1.94 20.0±0.03µs ? ?/sec 1.00 10.3±0.01µs ? ?/sec
date_part_nanosecond_ns_1000 2.01 18.4±0.02µs ? ?/sec 1.00 9.2±0.01µs ? ?/sec
date_part_nanosecond_ns_nulls_1000 1.00 9.1±0.01µs ? ?/sec
date_part_nanosecond_ns_tz_1000 2.40 91.5±0.25µs ? ?/sec 1.00 38.1±0.14µs ? ?/sec
date_part_nanosecond_s_1000 1.15 8.3±0.01µs ? ?/sec 1.00 7.2±0.00µs ? ?/sec
date_part_nanosecond_time32_ms_1000 2.03 2.7±0.01µs ? ?/sec 1.00 1318.3±5.64ns ? ?/sec
date_part_nanosecond_time64_ns_1000 2.36 2.6±0.01µs ? ?/sec 1.00 1099.0±2.59ns ? ?/sec
date_part_nanosecond_time64_ns_nulls_1000 1.00 1601.6±3.25ns ? ?/sec
date_part_nanosecond_us_1000 1.98 19.1±0.02µs ? ?/sec 1.00 9.6±0.03µs ? ?/sec
date_part_second_date32_1000 1.00 339.3±2.10ns ? ?/sec 1.04 353.6±3.67ns ? ?/sec
date_part_second_date64_1000 1.00 10.3±0.01µs ? ?/sec 1.00 10.3±0.02µs ? ?/sec
date_part_second_ms_1000 1.00 10.2±0.01µs ? ?/sec 1.01 10.3±0.02µs ? ?/sec
date_part_second_ns_1000 1.00 9.6±0.01µs ? ?/sec 1.00 9.6±0.03µs ? ?/sec
date_part_second_s_1000 1.00 7.7±0.02µs ? ?/sec 1.01 7.7±0.03µs ? ?/sec
date_part_second_us_1000 1.00 9.7±0.01µs ? ?/sec 1.01 9.9±0.01µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**datetime_expressions — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 505.1s |
| Peak memory | 34.3 MiB |
| Avg memory | 27.8 MiB |
| CPU user | 550.5s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
**datetime_expressions — branch**
| Metric | Value |
|--------|-------|
| Wall time | 600.1s |
| Peak memory | 34.8 MiB |
| Avg memory | 26.6 MiB |
| CPU user | 626.6s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4365941706
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -891,6 +866,38 @@ mod tests {
}
}
+ /// `epoch` is NULL only for a NULL input, but the integer parts can be NULL
+ /// for a non-null input, so they are always nullable.
+ #[test]
+ fn return_field_nullability() {
Review Comment:
The integer-part cases here are already exercised end to end by the non-null reproducers at the end of the slt. The ones from #26685 fail on `main` with `declared as non-nullable`. The only new information is that `epoch` follows the input. Could this move to the slt with the existing `DESCRIBE` idiom (see `describe.slt`)? For example `DESCRIBE SELECT extract(epoch from ts) AS e, extract(second from ts) AS s FROM (VALUES (timestamp '2020-01-01')) t(ts)`, once with and once without a NULL row. That shows the declared nullability directly.
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
+ // arrow's sub-second parts of intervals and durations already include
+ // the whole seconds
+ (
+ Interval(ArrowIntervalUnit::DayTime | ArrowIntervalUnit::MonthDayNano)
+ | Duration(_),
+ _,
+ ) => date_part(array, part)?,
+ (_, DatePart::Millisecond) => {
+ seconds_with_fraction::<Int32Type, 1_000>(array, part)?
+ }
+ (_, DatePart::Microsecond) => {
+ seconds_with_fraction::<Int32Type, 1_000_000>(array, part)?
+ }
+ (_, DatePart::Nanosecond) => {
+ seconds_with_fraction::<Int64Type, 1_000_000_000>(array, part)?
+ }
+ (_, part) => return internal_err!("{part} is not a sub-second part"),
+ })
+}
+/// Combine arrow's seconds of the minute with its fraction of the second in
+/// `part` units, for types where arrow returns them separately (timestamps
+/// and times).
+fn seconds_with_fraction<O, const PER_SECOND: i64>(
+ array: &dyn Array,
+ part: DatePart,
+) -> Result<ArrayRef>
+where
+ O: ArrowPrimitiveType,
+ i64: AsPrimitive<O::Native>,
+{
let secs = date_part(array, DatePart::Second)?;
// This assumes array is primitive and not a dictionary
let secs = as_int32_array(secs.as_ref())?;
- let subsecs = date_part(array, DatePart::Nanosecond)?;
- let subsecs = as_int32_array(subsecs.as_ref())?;
-
- // Special case where there are no nulls.
- if subsecs.null_count() == 0 {
- let r: Int64Array = binary(secs, subsecs, |secs, subsecs| {
- (secs as i64) * 1_000_000_000 + (subsecs as i64)
- })?;
- Ok(Arc::new(r))
- } else {
- // Nulls in secs are preserved, nulls in subsecs are treated as zero to account for the case
- // where the number of nanoseconds overflows.
- let r: Int64Array = secs
- .iter()
- .zip(subsecs)
- .map(|(secs, subsecs)| {
- secs.map(|secs| {
- let subsecs = subsecs.unwrap_or(0);
- (secs as i64) * 1_000_000_000 + (subsecs as i64)
- })
- })
- .collect();
- Ok(Arc::new(r))
- }
+ let fraction = date_part(array, part)?;
Review Comment:
For timestamps and times this makes three passes. Two `date_part` calls each run a chrono conversion per row, plus a timezone lookup per row and a timezone string parse per call when a timezone is set. Then `binary` allocates a third values buffer and a third bitmap for the null union. This matches the bench, where timestamps stay at about 1.1x while intervals and durations gain 3.6x to 41x.
`timezone_part_typed` above already has the one-pass shape. One `unary_opt` over `as_datetime_with_timezone::<T>` (or `as_datetime::<T>` without a timezone), computing `dt.second() * PER_SECOND + dt.nanosecond() / (1_000_000_000 / PER_SECOND)`, would do one conversion per row and one allocation. `main` already used two passes, so a follow-up is fine. I have not measured it. The `ns_tz` case you added is where I'd expect it to show most.
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
Review Comment:
These two arms can be one:
```rust
(Duration(_), DatePart::Nanosecond) => {
cast(&cast(array, &Duration(Nanosecond))?, &DataType::Int64)?
}
```
arrow's `cast` returns the input as is when the types already match. It reinterprets `Duration(Nanosecond)` as `Int64` without copying, and it already turns overflow into NULL for the unit change. That removes an arm and the repeated `as_duration_nanosecond_array(..)?.reinterpret_cast::<Int64Type>()` pair.
##########
datafusion/functions/src/datetime/date_part.rs:
##########
@@ -564,48 +504,83 @@ fn epoch(array: &dyn Array) -> Result<ArrayRef> {
Ok(Arc::new(f))
}
-/// Invoke [`date_part`] on an `array` (e.g. Timestamp) and convert the
-/// result to a total number of nanoseconds as an Int64 array.
+/// Return the seconds of `array`, with their fraction, as a number of
+/// milliseconds, microseconds or nanoseconds (`part`): the seconds of the
+/// minute for timestamps, times and intervals, and the total length for
+/// durations.
///
-/// This returns an Int64 rather than Int32 because there 1 billion
+/// `nanosecond` returns an Int64 rather than Int32 because there are 1 billion
/// `nanosecond`s in each second, so representing up to 60 seconds as
/// nanoseconds can be values up to 60 billion, which does not fit in Int32.
-fn seconds_ns(array: &dyn Array) -> Result<ArrayRef> {
- // Fast path for Date32 and Date64 - no nanoseconds
- if array.data_type() == &Date32 || array.data_type() == &Date64 {
- return Ok(Arc::new(Int64Array::from_iter_values_with_nulls(
- repeat_n(0, array.len()),
- array.nulls().cloned(),
- )));
- }
+fn subsecond_part(array: &dyn Array, part: DatePart) -> Result<ArrayRef> {
+ let len = array.len();
+ let nulls = array.nulls().cloned();
+ Ok(match (array.data_type(), part) {
+ // Fast path for Date32 and Date64 - no seconds
+ (Date32 | Date64, DatePart::Nanosecond) => {
+ Arc::new(Int64Array::new(vec![0; len].into(), nulls))
+ }
+ (Date32 | Date64, _) => Arc::new(Int32Array::new(vec![0; len].into(), nulls)),
+ // arrow's `date_part(Nanosecond)` computes these, but returns Int32,
+ // which overflows past about 2.1 seconds
+ (Interval(ArrowIntervalUnit::DayTime), DatePart::Nanosecond) => Arc::new(
+ as_interval_dt_array(array)?
+ .unary::<_, Int64Type>(|x| (x.milliseconds % 60_000) as i64 * 1_000_000),
+ ),
+ (Interval(ArrowIntervalUnit::MonthDayNano), DatePart::Nanosecond) => Arc::new(
+ as_interval_mdn_array(array)?
+ .unary::<_, Int64Type>(|x| x.nanoseconds % 60_000_000_000),
+ ),
+ (Duration(Nanosecond), DatePart::Nanosecond) => {
+ Arc::new(as_duration_nanosecond_array(array)?.reinterpret_cast::<Int64Type>())
+ }
+ // arrow's cast returns NULL on overflow, like its `date_part`
+ (Duration(_), DatePart::Nanosecond) => Arc::new(
+ as_duration_nanosecond_array(&cast(array, &Duration(Nanosecond))?)?
+ .reinterpret_cast::<Int64Type>(),
+ ),
+ // arrow's sub-second parts of intervals and durations already include
+ // the whole seconds
+ (
+ Interval(ArrowIntervalUnit::DayTime | ArrowIntervalUnit::MonthDayNano)
+ | Duration(_),
+ _,
+ ) => date_part(array, part)?,
Review Comment:
Could this be `Interval(_) | Duration(_)`? Today `Interval(YearMonth)` with `millisecond` or `microsecond` falls through to `seconds_with_fraction`, whose doc says it is for timestamps and times. It then fails on its first `date_part(.., Second)`, so `extract(millisecond ..)` reports `Second does not support: Interval(YearMonth)` (the slt at L1288 and L1291 pins this). arrow's `YearMonth` kernel rejects every sub-second part with the requested name, so the message would be accurate. One of those two `query error` cases would then be enough.
##########
datafusion/sqllogictest/test_files/datetime/date_part.slt:
##########
@@ -1289,6 +1419,124 @@ SELECT extract(day from arrow_cast(NULL, 'Duration(Second)'))
----
NULL
+# nanosecond on a duration is its total length in nanoseconds
+query IIII
+SELECT extract(nanosecond from arrow_cast(1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(1500000, 'Duration(Microsecond)')),
+ extract(nanosecond from arrow_cast(1500, 'Duration(Millisecond)')),
+ extract(nanosecond from arrow_cast(2, 'Duration(Second)'))
+----
+1500000000 1500000000 1500000000 2000000000
+
+# around i32::MAX nanoseconds
+query III
+SELECT extract(nanosecond from arrow_cast(2147483647, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(2147483648, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(62500000000, 'Duration(Nanosecond)'))
+----
+2147483647 2147483648 62500000000
+
+query II
+SELECT extract(nanosecond from arrow_cast(-1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(-62500, 'Duration(Millisecond)'))
+----
+-1500000000 -62500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast(column1, 'Duration(Millisecond)'))
+FROM (VALUES (1500), (NULL), (62500), (NULL), (-2))
+----
+1500000000
+NULL
+62500000000
+NULL
+-2000000
+
+query II
+SELECT extract(nanosecond from arrow_cast(9223372036854775807, 'Duration(Nanosecond)')),
+ extract(nanosecond from arrow_cast(-9223372036854775808, 'Duration(Nanosecond)'))
+----
+9223372036854775807 -9223372036854775808
+
+# values that overflow are NULL, like the other parts of a duration
+query II
+SELECT extract(nanosecond from arrow_cast(column1, 'Duration(Second)')),
+ extract(nanosecond from arrow_cast(column2, 'Duration(Millisecond)'))
+FROM (VALUES (9223372036, -9223372036854), (9223372037, -9223372036855), (NULL, NULL))
+----
+9223372036000000000 -9223372036854000000
+NULL NULL
+NULL NULL
+
+# millisecond and microsecond on a duration are its total length in that unit
+query II
+SELECT extract(millisecond from arrow_cast(5500000000, 'Duration(Nanosecond)')),
+ extract(microsecond from arrow_cast(5500000000, 'Duration(Nanosecond)'))
+----
+5500 5500000
+
+# truncates toward zero, like the other parts of a duration
+query III
+SELECT extract(millisecond from arrow_cast(-1500, 'Duration(Microsecond)')),
+ extract(millisecond from arrow_cast(-1500000, 'Duration(Nanosecond)')),
+ extract(second from arrow_cast(-1500, 'Duration(Millisecond)'))
+----
+-1 -1 -1
+
+query I
+SELECT extract(microsecond from arrow_cast(column1, 'Duration(Nanosecond)'))
+FROM (VALUES (5500000000), (NULL), (-62500000000), (NULL))
+----
+5500000
+NULL
+-62500000
+NULL
+
+# around i32::MAX
+query II
+SELECT extract(millisecond from arrow_cast(2147483647, 'Duration(Millisecond)')),
+ extract(microsecond from arrow_cast(2147, 'Duration(Second)'))
+----
+2147483647 2147000000
+
+# values that overflow Int32 are NULL, like the other parts of a duration
+query II
+SELECT extract(millisecond from arrow_cast(column1, 'Duration(Millisecond)')),
+ extract(microsecond from arrow_cast(column2, 'Duration(Second)'))
+FROM (VALUES (2147483648, 2148), (-2147483649, -2148), (NULL, NULL))
+----
+NULL NULL
+NULL NULL
+NULL NULL
+
+# Reproducer from https://github.com/apache/datafusion/issues/14738: 2.4e9 microseconds don't fit in Int32
+query I
+SELECT date_part('microsecond', timestamp '1970-01-01T00:40:00' - timestamp '1970-01-01T00:00:00')
+----
Review Comment:
This is the query from #14738, which asks for the Postgres result. In Postgres `timestamp - timestamp` is an interval and `microsecond` is the seconds field within the minute, so the issue expects 0. Here the difference is a `Duration`, and `microsecond` is its total length. Any duration longer than about 35.8 minutes is then NULL, and `millisecond` is NULL past about 24.8 days.
That matches how arrow treats the other `Duration` parts, so I think it is defensible. But it is a third answer to the issue, after the wrapped value and the panic. Should this be `Part of #14738`, with the semantics question left open? Or is NULL the agreed outcome? If total length is the intended meaning, a follow-up could return `Int64` for `Duration` input, since `return_field_from_args` sees the input type. That would avoid most of these NULLs.
##########
datafusion/spark/src/function/datetime/date_part.rs:
##########
@@ -81,10 +81,10 @@ impl ScalarUDFImpl for SparkDatePart {
internal_err!("Use return_field_from_args in this case instead.")
}
- fn return_field_from_args(&self, args: ReturnFieldArgs) -> Result<FieldRef> {
- let nullable = args.arg_fields.iter().any(|f| f.is_nullable());
-
- Ok(Arc::new(Field::new(self.name(), DataType::Int32, nullable)))
+ fn return_field_from_args(&self, _args: ReturnFieldArgs) -> Result<FieldRef> {
+ // Nullable like the integer parts of `date_part`, which this simplifies
+ // to, because they can be NULL for a non-null input
+ Ok(Arc::new(Field::new(self.name(), DataType::Int32, true)))
}
Review Comment:
With the input no longer read, this override now does exactly what the trait default does. `ScalarUDFImpl::return_field_from_args` builds `Field::new(self.name(), self.return_type(..)?, true)`. So `return_type` could return `Ok(DataType::Int32)` and the override could go.
Another option is to delegate to `datafusion_functions::datetime::date_part().return_field_from_args(args)`, which keeps the two matching by construction. That would also fix a mismatch this PR does not touch. `nanosecond` and `epoch` pass through `simplify` unchanged, but Spark declares `Int32` while `date_part` returns `Int64` and `Float64`.
Either way, the new test checks `Int32` and nullable directly and then again against `date_part`. The comparison alone is enough. Since `DESCRIBE` shows the planned schema before simplification, it could also be a `DESCRIBE SELECT date_part('SECOND', ts) ...` query in `spark/datetime/date_part.slt` instead of a unit test.
##########
datafusion/sqllogictest/test_files/datetime/date_part.slt:
##########
@@ -1143,24 +1177,120 @@ create table t (id int, i interval) as values
(4, interval '8 months'),
(5, NULL);
-query III
+query IIII
select
id,
extract(second from i),
- extract(month from i)
+ extract(month from i),
+ extract(nanosecond from i)
from t
order by id;
----
-0 0 5
-1 0 3
-2 0 0
-3 2 0
-4 0 8
-5 NULL NULL
+0 0 5 10
+1 0 3 0
+2 0 0 2000000
+3 2 0 2000000000
+4 0 8 0
+5 NULL NULL NULL
statement ok
drop table t;
+# nanosecond on an interval is the seconds field (within the minute) in nanoseconds,
+# like microsecond and millisecond
+query III
+SELECT extract(nanosecond from interval '1.5 seconds'),
+ extract(microsecond from interval '1.5 seconds'),
+ extract(millisecond from interval '1.5 seconds')
+----
+1500000000 1500000 1500
+
+# past i32::MAX nanoseconds
+query I
+SELECT extract(nanosecond from interval '62.5 seconds')
+----
+2500000000
+
+query I
+SELECT extract(nanosecond from interval '59.999999999 seconds')
+----
+59999999999
+
+query I
+SELECT extract(nanosecond from interval '1 day 2 hours 3 minutes')
+----
+0
+
+query I
+SELECT extract(nanosecond from interval '-1.5 seconds')
+----
+-1500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast('5.1 seconds', 'Interval(DayTime)'))
+----
+5100000000
+
+query I
+SELECT extract(nanosecond from arrow_cast('62.5 seconds', 'Interval(DayTime)'))
+----
+2500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast(NULL, 'Interval(DayTime)'))
+----
+NULL
+
+# millisecond and microsecond keep the fraction past i32::MAX nanoseconds
+query IIII
+SELECT extract(millisecond from interval '62.5 seconds'),
+ extract(microsecond from interval '62.5 seconds'),
+ extract(microsecond from interval '5.5 seconds'),
+ extract(microsecond from interval '59.999999 seconds')
+----
+2500 2500000 5500000 59999999
+
+query II
+SELECT extract(millisecond from interval '-62.5 seconds'),
+ extract(microsecond from interval '-62.5 seconds')
+----
+-2500 -2500000
+
+query II
+SELECT extract(millisecond from arrow_cast('62.5 seconds', 'Interval(DayTime)')),
+ extract(microsecond from arrow_cast('62.5 seconds', 'Interval(DayTime)'))
+----
+2500 2500000
+
+# Reproducer from https://github.com/apache/datafusion/issues/26678
+query III
+SELECT extract(nanosecond from arrow_cast(1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from interval '1.5 seconds'),
+ extract(microsecond from interval '1.5 seconds')
+----
+1500000000 1500000000 1500000
Review Comment:
These repeat queries added above. Column 1 is L1424, and columns 2 and 3 are L1201-L1202. In the next reproducer, the first column is L1245. Could the issue links go on those queries instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384408812
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384376483)
**Instance:** `c4-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6384376483-3833-nclfp 6.12.105+ #1 SMP Sat Sep 5 07:57:24 UTC 2026 x86_64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
resources:
arch: "amd64"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#issuecomment-6384401223
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26686#issuecomment-6384372592)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6384372592-3830-gss5l 6.12.105+ #1 SMP Sat Sep 5 07:54:11 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix-extract-nanosecond-duration-interval (48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f) to c920a5b92df0172b23026c3773dc7cd0fbdb8ce6 [diff](https://github.com/apache/datafusion/compare/c920a5b92df0172b23026c3773dc7cd0fbdb8ce6..48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f)
<details><summary>Run configuration</summary>
```yaml
run benchmark datetime_expressions
env:
BENCH_FILTER: "date_part_(second|millisecond|microsecond|nanosecond)_"
baseline:
ref: "c920a5b92df0172b23026c3773dc7cd0fbdb8ce6"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: date_part sub-second parts and result nullability, reusing arrow kernels [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #26686:
URL: https://github.com/apache/datafusion/pull/26686#discussion_r4367251970
##########
datafusion/sqllogictest/test_files/datetime/date_part.slt:
##########
@@ -1143,24 +1177,120 @@ create table t (id int, i interval) as values
(4, interval '8 months'),
(5, NULL);
-query III
+query IIII
select
id,
extract(second from i),
- extract(month from i)
+ extract(month from i),
+ extract(nanosecond from i)
from t
order by id;
----
-0 0 5
-1 0 3
-2 0 0
-3 2 0
-4 0 8
-5 NULL NULL
+0 0 5 10
+1 0 3 0
+2 0 0 2000000
+3 2 0 2000000000
+4 0 8 0
+5 NULL NULL NULL
statement ok
drop table t;
+# nanosecond on an interval is the seconds field (within the minute) in nanoseconds,
+# like microsecond and millisecond
+query III
+SELECT extract(nanosecond from interval '1.5 seconds'),
+ extract(microsecond from interval '1.5 seconds'),
+ extract(millisecond from interval '1.5 seconds')
+----
+1500000000 1500000 1500
+
+# past i32::MAX nanoseconds
+query I
+SELECT extract(nanosecond from interval '62.5 seconds')
+----
+2500000000
+
+query I
+SELECT extract(nanosecond from interval '59.999999999 seconds')
+----
+59999999999
+
+query I
+SELECT extract(nanosecond from interval '1 day 2 hours 3 minutes')
+----
+0
+
+query I
+SELECT extract(nanosecond from interval '-1.5 seconds')
+----
+-1500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast('5.1 seconds', 'Interval(DayTime)'))
+----
+5100000000
+
+query I
+SELECT extract(nanosecond from arrow_cast('62.5 seconds', 'Interval(DayTime)'))
+----
+2500000000
+
+query I
+SELECT extract(nanosecond from arrow_cast(NULL, 'Interval(DayTime)'))
+----
+NULL
+
+# millisecond and microsecond keep the fraction past i32::MAX nanoseconds
+query IIII
+SELECT extract(millisecond from interval '62.5 seconds'),
+ extract(microsecond from interval '62.5 seconds'),
+ extract(microsecond from interval '5.5 seconds'),
+ extract(microsecond from interval '59.999999 seconds')
+----
+2500 2500000 5500000 59999999
+
+query II
+SELECT extract(millisecond from interval '-62.5 seconds'),
+ extract(microsecond from interval '-62.5 seconds')
+----
+-2500 -2500000
+
+query II
+SELECT extract(millisecond from arrow_cast('62.5 seconds', 'Interval(DayTime)')),
+ extract(microsecond from arrow_cast('62.5 seconds', 'Interval(DayTime)'))
+----
+2500 2500000
+
+# Reproducer from https://github.com/apache/datafusion/issues/26678
+query III
+SELECT extract(nanosecond from arrow_cast(1500000000, 'Duration(Nanosecond)')),
+ extract(nanosecond from interval '1.5 seconds'),
+ extract(microsecond from interval '1.5 seconds')
+----
+1500000000 1500000000 1500000
Review Comment:
> Could the issue links go on those queries instead?
Done. The links are on the [interval](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1281-L1289) and [duration](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1516-L1524) queries for #26678, and on the [interval](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1355-L1363) and [duration](https://github.com/apache/datafusion/blob/48ff4d8da0d3a990f6ef216e13ef4e1f5cbe758f/datafusion/sqllogictest/test_files/datetime/date_part.slt#L1566-L1572) queries for the #14738 comment. The duration query now uses 2.5 s, so it covers the second column of that reproducer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org