You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2026/06/23 09:29:00 UTC

[jira] [Resolved] (SPARK-56769) Add fast path for date_trunc WEEK/MONTH/QUARTER/YEAR

     [ https://issues.apache.org/jira/browse/SPARK-56769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Max Gekk resolved SPARK-56769.
------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 55736
[https://github.com/apache/spark/pull/55736]

> Add fast path for date_trunc WEEK/MONTH/QUARTER/YEAR
> ----------------------------------------------------
>
>                 Key: SPARK-56769
>                 URL: https://issues.apache.org/jira/browse/SPARK-56769
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Rito Takeuchi
>            Assignee: Rito Takeuchi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> h2. Background
> `DateTimeUtils.truncTimestamp` for the WEEK / MONTH / QUARTER / YEAR levels currently routes through:
> {code:scala}
> case _ => // Try to truncate date levels
>   val dDays = microsToDays(micros, zoneId)
>   daysToMicros(truncDate(dDays, level), zoneId)
> {code}
> `microsToDays` allocates `Instant` + `ZonedDateTime` + `LocalDate` per row; `daysToMicros` allocates `LocalDate` + `ZonedDateTime` + `Instant`. `truncDate` itself allocates one more `LocalDate` for MONTH/YEAR (in `getDayOfMonth` / `getDayInYear`) and *two* for QUARTER (the existing implementation goes through `IsoFields.DAY_OF_QUARTER`, which is a `TemporalAdjuster` that produces a fresh `LocalDate`). The result is 167-218 ns/row on JDK 17 GH Actions runners.
> SPARK-56663 introduced the offset-arithmetic + DST-equality-guard pattern for the time-level units (MINUTE / HOUR / DAY) and confirmed the pattern is sound for any unit that evenly divides {{MICROS_PER_DAY}}. The date-level branch is a natural extension.
> h2. Proposal
> The framework for offset-based truncation -- resolve offset once, apply, truncate in the local frame, re-apply, DST guard, fall back on DST-cross or arithmetic overflow -- is identical for every level above SECOND. Only the "truncate in local frame" step varies. This PR inlines SPARK-56663's `truncToUnitFast` together with the new date-level path directly into `truncTimestamp`, and keeps a single private `truncTimestampSlow` as a complete reference implementation that the fast path falls back to:
> {code:scala}
> def truncTimestamp(micros: Long, level: Int, zoneId: ZoneId): Long = {
>   // MICROSECOND / MILLISECOND / SECOND short-circuits (no zone work).
>   // Offset arithmetic for every other level.
>   // DST guard, fallback to truncTimestampSlow.
> }
> private def truncTimestampSlow(micros: Long, level: Int, zoneId: ZoneId): Long
> {code}
> The local-frame truncation step is the only thing the fast path branches on:
> * MICROSECOND / MILLISECOND / SECOND -- pure UTC `floorMod` (zone offsets have at most second precision per `java.time.ZoneOffset`; no zone information needed).
> * MINUTE / HOUR / DAY -- shifted-local `floorMod` against the unit micros.
> * WEEK / MONTH / QUARTER / YEAR -- compute local epoch-day by integer division, run [[truncDate]] in the local-day frame, multiply back to local micros.
> Everything else (offset resolve via `rules.getOffset`, `addExact` / `subtractExact`, DST guard via offset-equality at the candidate, slow-path fallback) is shared.
> Also rewrite `TRUNC_TO_QUARTER` from `IsoFields.DAY_OF_QUARTER` (a `TemporalAdjuster` that produces a fresh `LocalDate`) to a direct `withMonth(firstMonthOfQuarter).withDayOfMonth(1)` chain on the existing `LocalDate`. Saves one allocation + the adjuster overhead per call.
> `truncTimestampSlow` covers every level explicitly so it serves as a self-contained reference implementation -- the fast path's correctness can be verified against it case-by-case.
> h2. Benchmark
> `DateTimeBenchmark` Truncation, wholestage on, ns/row, on a 12th Gen Intel i7-1260P (master = pre-SPARK-56663):
> || level || master baseline || this PR || speedup ||
> | WEEK    | 165.2 |  78.2 | 2.11x |
> | MONTH   | 181.9 |  92.2 | 1.97x |
> | MM      | 182.2 |  92.5 | 1.97x |
> | MON     | 182.9 |  92.7 | 1.97x |
> | QUARTER | 216.8 | 108.8 | 1.99x |
> | YEAR    | 205.2 |  96.7 | 2.12x |
> | YYYY    | 205.8 |  96.9 | 2.12x |
> | YY      | 206.3 |  96.0 | 2.15x |
> Time-level units (MINUTE / HOUR / DAY / SECOND) and `trunc(date, ...)` are unchanged within noise; this PR's hot path for those levels is byte-identical to SPARK-56663 after the unification.
> h2. Out of scope
> * `trunc(date, ...)` (date input, no zoneId) -- this PR only changes the `timestamp -> date_trunc` flow. The `TruncDate` expression bypasses `truncTimestamp`; the only change visible to it is the `TRUNC_TO_QUARTER` rewrite (which `trunc(date, ...)` doesn't use in the benchmark today).
> * MICROSECOND / MILLISECOND / SECOND / MINUTE / HOUR / DAY -- handled by SPARK-56663. The unification in this PR inlines the existing fast path into `truncTimestamp` but does not change its semantics or measured perf.
> * Per-instance offset cache -- a separate optimization that would amortize the {{rules.getOffset}} call across rows. Would benefit both this PR's and SPARK-56663's paths. Out of scope here.
> * Integer-only calendar arithmetic (Hinnant-style) -- would eliminate the remaining `LocalDate` allocation inside `truncDate` for MONTH/YEAR and push date-level units to the same floor as time-level units. Out of scope here.
> h2. Related
> * SPARK-56663 - introduced the offset-arithmetic fast path for MIN/HR/DAY; this PR extends the same pattern to the date-level units and inlines both paths into a single implementation.
> * SPARK-33404 - introduced the slow path that this family of changes is recovering from.
> * SPARK-30766 / SPARK-30857 - the DST-correctness invariants from these fixes are preserved here via the offset-equality guard.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org