You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/02/20 14:55:27 UTC
[spark] branch master updated: [SPARK-26900][SQL] Simplify
truncation to quarter of year
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 331ac60 [SPARK-26900][SQL] Simplify truncation to quarter of year
331ac60 is described below
commit 331ac60f2818f191efb583ba4e73467f2a9d7d17
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Wed Feb 20 08:55:08 2019 -0600
[SPARK-26900][SQL] Simplify truncation to quarter of year
## What changes were proposed in this pull request?
In the PR, I propose to simplify timestamp truncation to quarter of year by using *java.time* API directly. The `LocalDate` instance can be truncation to quarter timestamp via adjusting by chrono field `IsoFields.DAY_OF_QUARTER`.
## How was this patch tested?
This was checked by existing test suite - `DateTimeUtilsSuite`.
Closes #23808 from MaxGekk/date-quarter-of-year.
Authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Sean Owen <se...@databricks.com>
---
.../org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 14 +++-----------
sql/core/benchmarks/DateTimeBenchmark-results.txt | 6 +++---
2 files changed, 6 insertions(+), 14 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index b537695..28fb6a9 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -719,17 +719,9 @@ object DateTimeUtils {
daysToMillis(prevMonday, timeZone)
case TRUNC_TO_QUARTER =>
val dDays = millisToDays(millis, timeZone)
- val month = getQuarter(dDays) match {
- case 1 => Month.JANUARY
- case 2 => Month.APRIL
- case 3 => Month.JULY
- case 4 => Month.OCTOBER
- }
- millis = daysToMillis(truncDate(dDays, TRUNC_TO_MONTH), timeZone)
- val instant = Instant.ofEpochMilli(millis)
- val localDateTime = LocalDateTime.ofInstant(instant, timeZone.toZoneId)
- val truncated = localDateTime.withMonth(month.getValue)
- truncated.atZone(timeZone.toZoneId).toInstant.toEpochMilli
+ val daysOfQuarter = LocalDate.ofEpochDay(dDays)
+ .`with`(IsoFields.DAY_OF_QUARTER, 1L).toEpochDay.toInt
+ daysToMillis(daysOfQuarter, timeZone)
case _ =>
// caller make sure that this should never be reached
sys.error(s"Invalid trunc level: $level")
diff --git a/sql/core/benchmarks/DateTimeBenchmark-results.txt b/sql/core/benchmarks/DateTimeBenchmark-results.txt
index 8bbe310..bc824af 100644
--- a/sql/core/benchmarks/DateTimeBenchmark-results.txt
+++ b/sql/core/benchmarks/DateTimeBenchmark-results.txt
@@ -324,12 +324,12 @@ date_trunc WEEK: Best/Avg Time(ms) Rate(M/s) Per Ro
date_trunc WEEK wholestage off 794 / 797 12.6 79.4 1.0X
date_trunc WEEK wholestage on 754 / 761 13.3 75.4 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc QUARTER: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc QUARTER wholestage off 5261 / 5271 1.9 526.1 1.0X
-date_trunc QUARTER wholestage on 5145 / 5151 1.9 514.5 1.0X
+date_trunc QUARTER wholestage off 1465 / 1467 6.8 146.5 1.0X
+date_trunc QUARTER wholestage on 1419 / 1423 7.0 141.9 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org